__ __ / \ / \ ______ _ _ __ __ __ / /\ / /\ \ / __ \ | | | | / \ \ \ / / / / / / \ \ | | | | | | | | / /\ \ \ / \ \ \ \ / / | |__| | | |__| | / ____ \ | | \ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_| \__/ \__/ \ \__ \___\ by Red Hat Build, Store, and Distribute your Containers Startup timestamp: Tue Jan 17 04:49:08 UTC 2023 Running all default registry services without migration Running init script '/quay-registry/conf/init/certs_install.sh' Installing extra certificates found in /quay-registry/conf/stack/extra_ca_certs directory Running init script '/quay-registry/conf/init/copy_config_files.sh' Running init script '/quay-registry/conf/init/d_validate_config_bundle.sh' Validating Configuration time="2023-01-17T04:49:09Z" level=debug msg="Validating AccessSettings" time="2023-01-17T04:49:09Z" level=debug msg="Validating ActionLogArchiving" time="2023-01-17T04:49:09Z" level=debug msg="Validating AppTokenAuthentication" time="2023-01-17T04:49:09Z" level=debug msg="Validating BitbucketBuildTrigger" time="2023-01-17T04:49:09Z" level=debug msg="bitbucket response: 400, {\"error\": \"invalid_grant\", \"error_description\": \"The specified code is not valid.\"}" time="2023-01-17T04:49:09Z" level=debug msg="Validating BuildManager" time="2023-01-17T04:49:09Z" level=debug msg="Validating Database" time="2023-01-17T04:49:09Z" level=debug msg="Scheme: postgresql" time="2023-01-17T04:49:09Z" level=debug msg="Host: quayregistry-quay-database:5432" time="2023-01-17T04:49:09Z" level=debug msg="Db: quayregistry-quay-database" time="2023-01-17T04:49:09Z" level=debug msg="Params: " time="2023-01-17T04:49:09Z" level=debug msg="Including params " time="2023-01-17T04:49:09Z" level=debug msg="Pinging database at postgresql://quayregistry-quay-database:A0wE7b-Mv8cs1PgSnHaCuX3YAbtMKF1Aodz4cvKHu5mmCSqpoTsvUnE8iRfsQoi6WlAQc6u7ZlMbl8dL@quayregistry-quay-database:5432/quayregistry-quay-database" plpgsql pg_trgm time="2023-01-17T04:49:09Z" level=debug msg="Validating DistributedStorage" time="2023-01-17T04:49:09Z" level=debug msg="Validating ElasticSearch" time="2023-01-17T04:49:09Z" level=debug msg="Validating Email" time="2023-01-17T04:49:09Z" level=debug msg="Validating GitHubBuildTrigger" time="2023-01-17T04:49:10Z" level=debug msg="github response: 200, \n\n\n\n\n\n\n\n \n \n \n \n \n \n \n \n\n\n\n \n \n \n \n \n \n\n\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n GitHub: Let’s build from here · GitHub\n\n\n\n \n\n \n\n\n\n \n\n \n \n\n \n\n \n \n \n \n \n\n\n\n \n\n \n\n\n\n\n \n\n \n\n \n\n \n \n \n \n \n \n \n \n \n \n \n\n\n \n\n\n\n \n\n \n\n\n \n \n \n \n\n \n\n \n \n \n\n \n\n \n \n \n\n\n\n \n \n\n\n \n\n \n\n \n\n \n \n \n\n\n\n\n\n \n\n \n\n \n
\n \n\n\n
\n Skip to content\n \n \n \n \n\n\n \n\n \n
\n \n\n
\n
\n \n \n \n\n \n\n
\n \n Sign up\n \n
\n\n
\n \n
\n
\n\n\n
\n
\n \n\n
\n
\n \n\n\n\n
\n
\n
\n \n
\n
\n\n
\n\n
\n \n Sign in\n \n
\n\n \n Sign up\n \n
\n
\n
\n
\n
\n\n
\n\n
\n\n\n\n\n\n\n\n
\n\n\n\n\n\n \n
\n\n\n \n \n\n\n\n\n\n\n \n
\n\n\n
\n
\n \n \n \n\n \"\"\n \n
\n\n
\n
\n \n\n
\n
\n \"\"\n\n
\n
\n \n \n\n \n
\n
\n\n
\n
\n\n
\n
\n \n
\n\n

\n Let’s build from here\n

\n\n

\n Harnessed for productivity. Designed for collaboration. Celebrated for built-in security. Welcome to the platform developers love.\n

\n\n
\n
\n
\n
\n \n
\n
\n \n
\n
\n \n \n
\n
\n
\n\n \n Start a free enterprise trial\n \n \n\n\n
\n\n
\n

Trusted by the world’s leading organizations ↘︎

\n\n
\n \"Stripe\n \"Pinterest\n
\n \"KPMG\n
\n \"Mercedes-Benz\n
\n \"P&G\n \"Telus\n
\n
\n
\n
\n
\n
\n\n
\n
\n
\n \n\n \n\n
\n Productivity\n Collaboration\n Security\n
\n\n
\n \n Start a free enterprise trial\n \n \n\n\n \n Sign up for GitHub\n \n \n\n\n
\n
\n
\n
\n\n
\n
\n \n
\n
\n
\n \n \n\n \n
\n\n
\n
\n\n
\n

Productivity

\n

\n Accelerate high-quality software development.\n Our platform drives innovation with tools that boost developer velocity.\n

\n
\n
\n\n
\n
\n
\n \n \n\n \n \n\n
\n
\n \n \n\n
mona-github-github-g59jpq2w5w7.github.dev
\n \n \n\n
\n
\n \n \n\n
\n
\n\n
\n
\n \n \n\n
\n\n
\n
\n
    \n
  • index.html
    • \n
  • script.js
    • \n
  • package.json
    • \n
\n\n
\n
\n
\n
1
\n
2
\n
3
\n
4
\n
5
\n
6
\n
7
\n
8
\n
9
\n
10
\n
11
\n
12
\n
\n\n
\n 
<div class="position-absolute width-full color-bg-default" style="bottom: -4rem;">
\n 
  <div class="container-xl p-responsive">
\n 
    <div class="d-flex flex-justify-center flex-lg-justify-end color-bg-default">
\n 
      <div class="col-8 col-sm-7 col-md-6 col-lg-5 position-relative z-2 right-lg-n12 events-none">
\n 
        <picture>
\n 
          <source srcset="astro-mona.webp" type="image/webp">
\n 
          <img src="astro-mona.svg" width="960" height="967" class="home-astro-mona width-full position-absolute bottom-0 height-auto" alt="Mona looking at GitHub activity across the globe">
\n 
        </picture>
\n 
      </div>
\n 
    </div>
\n 
  </div>
\n 
</div>
\n\n
\n
\n\n\n
\n
\n\n
\n
    \n
  • Terminal
    • \n
  • Output
    • \n
  • Problems
    • \n
  • Debug Console
    • \n
\n\n
\n
\n [09:43:36]\n Starting\n 'watch-extension:vscode-api-tests'\n ...\n
\n
\n [09:43:36]\n Finished\n 'clean-extension:typescript-language-features'\n after\n 248 ms\n
\n
\n [09:43:36]\n Starting\n 'watch-extension:typescript-language-features'\n ...\n
\n
\n [09:43:36]\n Finished\n 'clean-extension:php-language-features'\n after\n 384 ms\n
\n
\n [09:43:36]\n Starting\n 'watch-extension:php-language-features'\n ...\n
\n
\n [09:43:40]\n Finished\n 'clean-extension:html-language-features-server'\n after\n 4.66 s\n
\n
\n [09:43:40]\n Starting\n 'watch-extension:html-language-features-server'\n ...\n
\n
\n [09:43:43]\n Finished\n 'clean-client'\n after\n 7.33 s\n
\n
\n [09:43:43]\n Starting\n 'watch-client'\n ...\n
\n
\n
\n
\n\n
\n
\n
\n \n
\n
\n \n
\n
\n \n
\n
\n\n
\n \n \n \n \"Planet\n\n
\n
\n
\n
\n\n\n
\n \"\"\n\n
\n
\n \"\"\n
\n\n
\n
\n
\n
\n \n \n \n \n

Codespaces offers a complete dev environment in seconds, so you can code, build, test, and open pull requests from any repo anywhere.

\n \n \n Check out Codespaces \n\n\n \n
\n
\n\n
\n
\n \n
\n
\n \"\"\n \"\"\n
\n
\n \"\"\n
\n
\n\n
\n
\n
\n
\n
\n
\n Did you know?\n
\n\n

22% increase

\n

in developer productivity
after three years with GitHub1

\n
\n
\n
\n
\n\n
\n
\n

GitHub Copilot is your AI pair programmer that empowers you to complete tasks 55% faster by turning natural language prompts into coding suggestions.

\n\n \n
\n \n
\n
\n \n\n \n
\n \n
\n\n
\n
\n
\n
1
\n
2
\n
3
\n
4
\n
5
\n
6
\n
7
\n
8
\n
\n\n
\n 
import matplotlib.pyplot as plt
\n
\n 
def draw_scatterplot(x_values, y_values):
\n 
    plt.scatter(x_values, y_values, s=20)
\n 
    plt.title("Scatter Plot")
\n 
    plt.xlabel("x values")
\n 
    plt.ylabel("y values")
\n 
    plt.show()
\n\n
\n \n\n Copilot\n
\n
\n
\n\n
\n
\n\n
\n
\n
\n
\n
\n \n
\n
\n
\n

GitHub Actions automates your build, test, and deployment workflow with simple and secure CI/CD.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n
\n
\n

GitHub Mobile fits your projects in your pocket, so you never miss a beat while on the go.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n\n
\n
\n
\n\n
\n
\n
\n
\n
\n \n
\n
\n
\n \n \n\n \n
\n\n
\n
\n\n
\n

Collaboration

\n

\n Supercharge collaboration.\n We provide unlimited repositories, best-in-class version control, and the world’s most powerful open source community—so your team can work more efficiently together.\n

\n
\n
\n\n \n \n \n \"Illustration\n\n\n
\n \"\"\n\n
\n
\n \"\"\n
\n\n
\n
\n
\n
\n \n \n \n \n

GitHub Issues and GitHub Projects supply flexible project management tools that adapt to your team alongside your code.

\n \n \n Explore GitHub Issues \n\n\n \n
\n
\n\n
\n
\n \n
\n
\n \"\"\n
\n
\n\n
\n
\n
\n
\n
\n
\n Did you know?\n
\n\n

80% reduction

\n

in onboarding time with GitHub1

\n
\n
\n
\n
\n\n
\n \n
\n
\n
\n

GitHub Discussions create space to ask questions and have open-ended conversations.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n
\n
\n

Pull requests allow real-time communication and collaboration about code changes.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n\n
\n
\n

GitHub Sponsors lets you support your favorite open source maintainers and projects.

\n\n
\n Invest with GitHub Sponsors \n\n
\n
\n \n
\n
\n
\n \n \"ESLint\n
ESLint
\n \n
\n
\n \n \"Nick\n
Nick DeJesus
\n \n
\n
\n \n \"kazuya\n
kazuya kawaguchi
\n \n
\n
\n \n \"sindresorhus\n
sindresorhus
\n \n
\n
\n \n \"Nikema\n
Nikema
\n \n
\n
\n \n \"curl\n
curl
\n \n
\n
\n \n \"Directus\n
Directus
\n \n
\n
\n \n \"CommandPost\n
CommandPost
\n \n
\n
\n \n \"Homebrew\n
Homebrew
\n \n
\n
\n \n \"chaynHQ\n
chaynHQ
\n \n
\n
\n \n \"Evan\n
Evan You
\n \n
\n
\n \n \"Samuel\n
Samuel
\n \n
\n
\n
\n
\n
\n
\n
\n\n
\n
\n
\n
\n \n
\n
\n
\n \n \n\n \n
\n\n
\n
\n\n
\n

Security

\n

\n Embed security into the developer workflow.\n With GitHub, developers can secure their code in minutes and organizations can automatically comply with regulations.\n

\n
\n
\n\n
\n
\n
cmake.yml
\n on: push\n
\n\n
\n
\n
\n
    \n
  • \n
    \n \n \n\n Build\n
    \n 1m 21s\n
    • \n
\n
\n
\n\n
\n
\n
\n
\n
\n\n
\n
\n Steps\n
\n
\n
    \n
  • \n
    \n \n \n\n Initialize CodeQL\n
    \n 1m 42s\n
    • \n
  • \n
    \n \n \n\n Autobuild\n
    \n 1m 24s\n
    • \n
  • \n
    \n \n \n\n Perform CodeQL Analyses\n
    \n 1m 36s\n
    • \n
\n
\n
\n
\n
\n\n
\n \"\"\n\n
\n
\n \"\"\n
\n\n
\n
\n
\n
\n \n \n \n \n

GitHub Advanced Security lets you gain visibility into your security posture, respond to threats proactively, and ship secure applications quickly.

\n \n \n Get GitHub Advanced Security \n\n\n \n
\n
\n\n
\n
\n \n
\n
\n \"\"\n
\n
\n\n
\n
\n
\n
\n
\n
\n Did you know?\n
\n\n

56 million projects

\n

fixed vulnerabilities with GitHub2

\n
\n
\n
\n
\n\n
\n
\n
\n

Secret scanning automatically looks for partner patterns and prevents fraudulent use of accidentally committed secrets.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n \n
\n
\n
\n

Dependabot makes it easy to find and fix vulnerable dependencies in your supply chain.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n
\n
\n

Code scanning is GitHub’s static code analysis tool that helps you remediate issues in your code.

\n\n \n
\n
\n \n \n \"\"\n
\n \n
\n
\n\n
\n\n
\n
\n\n
\n \n \n
\n
\n
\n
\n \n \n \n \n

The place for anyone from anywhere to build anything

\n

Whether you’re scaling your startup or just learning how to code, GitHub is your home. Join the world’s largest developer platform to build the innovations that empower humanity. Let’s build from here.

\n \n
\n\n
\n \n Sign up for GitHub\n \n \n\n\n \n Start a free enterprise trial\n \n \n\n\n
\n\n
\n\n
\n
\n\n
\n\n
\n
\n
\n \n \n \n\n \n\n
\n
\n\n \n
\n\n\n \n \n \n \"\"\n\n\n
\n \"Mona\n
\n\n \n\n\n \n
\n
\n \n
\n\n\n\n \n\n \n\n\n\n\n \n\n
\n \n \n\n \n \n
\n \n\n
\n
\n
\n
\n\n \n\n\n\n\n \n\n
\n \n\n\n" time="2023-01-17T04:49:10Z" level=debug msg="Validating GitHubLogin" time="2023-01-17T04:49:10Z" level=debug msg="Validating GitLabBuildTrigger" time="2023-01-17T04:49:10Z" level=debug msg="Validating GoogleLogin" time="2023-01-17T04:49:10Z" level=debug msg="Validating HostSettings" time="2023-01-17T04:49:10Z" level=debug msg="Validating JWTAuthentication" time="2023-01-17T04:49:10Z" level=debug msg="Validating LDAP" time="2023-01-17T04:49:10Z" level=debug msg="Validating OIDC" time="2023-01-17T04:49:10Z" level=debug msg="Validating QuayDocumentation" time="2023-01-17T04:49:10Z" level=debug msg="Validating Redis" time="2023-01-17T04:49:10Z" level=debug msg="Address: quayregistry-quay-redis:6379" time="2023-01-17T04:49:10Z" level=debug msg="Username: " time="2023-01-17T04:49:10Z" level=debug msg="Password Len: 0" time="2023-01-17T04:49:10Z" level=debug msg="Ssl: " time="2023-01-17T04:49:10Z" level=debug msg="Address: quayregistry-quay-redis:6379" time="2023-01-17T04:49:10Z" level=debug msg="Username: " time="2023-01-17T04:49:10Z" level=debug msg="Password Len: 0" time="2023-01-17T04:49:10Z" level=debug msg="Ssl: " time="2023-01-17T04:49:10Z" level=debug msg="Validating RepoMirror" time="2023-01-17T04:49:10Z" level=debug msg="Validating SecurityScanner" time="2023-01-17T04:49:10Z" level=debug msg="Validating TeamSyncing" time="2023-01-17T04:49:10Z" level=debug msg="Validating TimeMachine" time="2023-01-17T04:49:10Z" level=debug msg="Validating UserVisibleSettings" +------------------------+-------+--------+ | Field Group | Error | Status | +------------------------+-------+--------+ | AccessSettings | - | 🟢 | +------------------------+-------+--------+ | ActionLogArchiving | - | 🟢 | +------------------------+-------+--------+ | AppTokenAuthentication | - | 🟢 | +------------------------+-------+--------+ | BitbucketBuildTrigger | - | 🟢 | +------------------------+-------+--------+ | BuildManager | - | 🟢 | +------------------------+-------+--------+ | Database | - | 🟢 | +------------------------+-------+--------+ | DistributedStorage | - | 🟢 | +------------------------+-------+--------+ | ElasticSearch | - | 🟢 | +------------------------+-------+--------+ | Email | - | 🟢 | +------------------------+-------+--------+ | GitHubBuildTrigger | - | 🟢 | +------------------------+-------+--------+ | GitHubLogin | - | 🟢 | +------------------------+-------+--------+ | GitLabBuildTrigger | - | 🟢 | +------------------------+-------+--------+ | GoogleLogin | - | 🟢 | +------------------------+-------+--------+ | HostSettings | - | 🟢 | +------------------------+-------+--------+ | JWTAuthentication | - | 🟢 | +------------------------+-------+--------+ | LDAP | - | 🟢 | +------------------------+-------+--------+ | OIDC | - | 🟢 | +------------------------+-------+--------+ | QuayDocumentation | - | 🟢 | +------------------------+-------+--------+ | Redis | - | 🟢 | +------------------------+-------+--------+ | RepoMirror | - | 🟢 | +------------------------+-------+--------+ | SecurityScanner | - | 🟢 | +------------------------+-------+--------+ | TeamSyncing | - | 🟢 | +------------------------+-------+--------+ | TimeMachine | - | 🟢 | +------------------------+-------+--------+ | UserVisibleSettings | - | 🟢 | +------------------------+-------+--------+ Running init script '/quay-registry/conf/init/nginx_conf_create.sh' Running init script '/quay-registry/conf/init/supervisord_conf_create.sh' Running init script '/quay-registry/conf/init/zz_boot.sh' 2023-01-17 04:49:12,088 INFO RPC interface 'supervisor' initialized 2023-01-17 04:49:12,088 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2023-01-17 04:49:12,088 INFO supervisord started with pid 7 2023-01-17 04:49:13,091 INFO spawned: 'stdout' with pid 58 2023-01-17 04:49:13,092 INFO spawned: 'blobuploadcleanupworker' with pid 59 2023-01-17 04:49:13,093 INFO spawned: 'builder' with pid 60 2023-01-17 04:49:13,094 INFO spawned: 'buildlogsarchiver' with pid 61 2023-01-17 04:49:13,096 INFO spawned: 'chunkcleanupworker' with pid 62 2023-01-17 04:49:13,096 INFO spawned: 'dnsmasq' with pid 63 2023-01-17 04:49:13,097 INFO spawned: 'expiredappspecifictokenworker' with pid 64 2023-01-17 04:49:13,099 INFO spawned: 'exportactionlogsworker' with pid 65 2023-01-17 04:49:13,100 INFO spawned: 'gcworker' with pid 66 2023-01-17 04:49:13,101 INFO spawned: 'globalpromstats' with pid 67 2023-01-17 04:49:13,103 INFO spawned: 'gunicorn-registry' with pid 68 2023-01-17 04:49:13,104 INFO spawned: 'gunicorn-secscan' with pid 69 2023-01-17 04:49:13,106 INFO spawned: 'gunicorn-web' with pid 70 2023-01-17 04:49:13,107 INFO spawned: 'logrotateworker' with pid 71 2023-01-17 04:49:13,108 INFO spawned: 'manifestbackfillworker' with pid 72 2023-01-17 04:49:13,110 INFO spawned: 'memcache' with pid 73 2023-01-17 04:49:13,111 INFO spawned: 'namespacegcworker' with pid 74 2023-01-17 04:49:13,112 INFO spawned: 'nginx' with pid 75 2023-01-17 04:49:13,134 INFO spawned: 'notificationworker' with pid 76 2023-01-17 04:49:13,136 INFO spawned: 'pushgateway' with pid 77 2023-01-17 04:49:13,137 INFO spawned: 'queuecleanupworker' with pid 78 2023-01-17 04:49:13,139 INFO spawned: 'repositoryactioncounter' with pid 79 2023-01-17 04:49:13,140 INFO spawned: 'repositorygcworker' with pid 80 2023-01-17 04:49:13,142 INFO spawned: 'securityscanningnotificationworker' with pid 81 2023-01-17 04:49:13,144 INFO spawned: 'securityworker' with pid 87 2023-01-17 04:49:13,145 INFO spawned: 'servicekey' with pid 88 2023-01-17 04:49:13,235 INFO spawned: 'storagereplication' with pid 89 2023-01-17 04:49:13,237 INFO spawned: 'teamsyncworker' with pid 90 2023-01-17 04:49:14,844 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,844 INFO success: blobuploadcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,844 INFO success: builder entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,844 INFO success: buildlogsarchiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,844 INFO success: chunkcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,844 INFO success: dnsmasq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,844 INFO success: expiredappspecifictokenworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,844 INFO success: exportactionlogsworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: gcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: globalpromstats entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: gunicorn-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: gunicorn-secscan entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: gunicorn-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: logrotateworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: manifestbackfillworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: memcache entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: namespacegcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: notificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: pushgateway entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: queuecleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: repositoryactioncounter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: repositorygcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: securityscanningnotificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: securityworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: servicekey entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: storagereplication entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2023-01-17 04:49:14,845 INFO success: teamsyncworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) dnsmasq stderr | dnsmasq: started, version 2.79 cachesize 150 dnsmasq stderr | dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dnsmasq stderr | dnsmasq: reading /etc/resolv.conf dnsmasq stderr | dnsmasq: using nameserver 172.30.0.10#53 dnsmasq stderr | dnsmasq: read /etc/hosts - 7 addresses nginx stdout | 2023/01/17 04:49:13 [warn] 75#0: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:40 nginx stdout | 2023/01/17 04:49:13 [warn] 75#0: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:67 nginx stdout | 2023/01/17 04:49:13 [warn] 75#0: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:91 nginx stderr | nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:40 nginx stderr | nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:67 nginx stderr | nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /quay-registry/conf/nginx/nginx.conf:91 pushgateway stderr | level=info ts=2023-01-17T04:49:13.441Z caller=main.go:85 msg="starting pushgateway" version="(version=, branch=, revision=)" pushgateway stderr | level=info ts=2023-01-17T04:49:13.441Z caller=main.go:86 build_context="(go=go1.17.12, user=, date=)" nginx stdout | 2023/01/17 04:49:13 [alert] 105#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 111#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 115#0: setpriority(-10) failed (13: Permission denied) pushgateway stderr | level=info ts=2023-01-17T04:49:13.635Z caller=main.go:139 listen_address=:9091 pushgateway stderr | level=info ts=2023-01-17T04:49:13.635Z caller=tls_config.go:191 msg="TLS is disabled." http2=false nginx stdout | 2023/01/17 04:49:13 [alert] 107#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 109#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 112#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 108#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 101#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 102#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 113#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 100#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 114#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 110#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 103#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 106#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:13 [alert] 104#0: setpriority(-10) failed (13: Permission denied) nginx stdout | 2023/01/17 04:49:22 [crit] 111#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.129.2.33:8443" nginx stdout | 2023/01/17 04:49:22 [crit] 111#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.1, server: _, request: "GET /health/instance HTTP/2.0", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.129.2.33:8443" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:49:22 +0000] "GET /health/instance HTTP/2.0" 502 157 "-" "kube-probe/1.24" (0.000 47 0.000 : 0.000) gunicorn-web stdout | 2023-01-17 04:49:28,136 [70] [INFO] [data.registry_model] =============================== gunicorn-web stdout | 2023-01-17 04:49:28,136 [70] [INFO] [data.registry_model] Using registry model `` gunicorn-web stdout | 2023-01-17 04:49:28,136 [70] [INFO] [data.registry_model] =============================== logrotateworker stdout | 2023-01-17 04:49:30,239 [71] [DEBUG] [__main__] Action log rotation worker not enabled; skipping gcworker stdout | 2023-01-17 04:49:30,240 [66] [DEBUG] [workers.worker] Scheduling worker. gcworker stdout | 2023-01-17 04:49:30,240 [66] [INFO] [apscheduler.scheduler] Scheduler started gunicorn-web stdout | 2023-01-17 04:49:30,247 [70] [DEBUG] [app] Loading default config. gunicorn-web stdout | 2023-01-17 04:49:30,247 [70] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml queuecleanupworker stdout | 2023-01-17 04:49:30,335 [78] [DEBUG] [workers.worker] Scheduling worker. queuecleanupworker stdout | 2023-01-17 04:49:30,336 [78] [INFO] [apscheduler.scheduler] Scheduler started gcworker stdout | 2023-01-17 04:49:30,347 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:49:30,347 [66] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:51.348605 queuecleanupworker stdout | 2023-01-17 04:49:30,434 [78] [DEBUG] [apscheduler.scheduler] Looking for jobs to run queuecleanupworker stdout | 2023-01-17 04:49:30,434 [78] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added gunicorn-web stdout | 2023-01-17 04:49:30,441 [70] [DEBUG] [app] Loaded config gunicorn-web stdout | 2023-01-17 04:49:30,442 [70] [INFO] [util.ipresolver] Loading AWS IP ranges from disk queuecleanupworker stdout | 2023-01-17 04:49:30,534 [78] [DEBUG] [workers.worker] First run scheduled for 2023-01-18 00:07:43.535392 gcworker stdout | 2023-01-17 04:49:30,434 [66] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added gunicorn-web stdout | 2023-01-17 04:49:30,541 [70] [DEBUG] [util.ipresolver] Building AWS IP ranges namespacegcworker stdout | 2023-01-17 04:49:30,747 [74] [DEBUG] [__main__] Starting namespace GC worker namespacegcworker stdout | 2023-01-17 04:49:31,042 [74] [DEBUG] [workers.worker] Scheduling worker. namespacegcworker stdout | 2023-01-17 04:49:31,043 [74] [INFO] [apscheduler.scheduler] Scheduler started namespacegcworker stdout | 2023-01-17 04:49:31,146 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:49:31,146 [74] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added namespacegcworker stdout | 2023-01-17 04:49:31,147 [74] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:41.148245 gcworker stdout | 2023-01-17 04:49:31,241 [66] [INFO] [apscheduler.scheduler] Added job "GarbageCollectionWorker._garbage_collection_repos" to job store "default" gcworker stdout | 2023-01-17 04:49:31,245 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:49:31,245 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:51.348605+00:00 (in 20.103136 seconds) notificationworker stdout | 2023-01-17 04:49:31,540 [76] [DEBUG] [workers.worker] Scheduling worker. notificationworker stdout | 2023-01-17 04:49:31,541 [76] [INFO] [apscheduler.scheduler] Scheduler started queuecleanupworker stdout | 2023-01-17 04:49:31,540 [78] [INFO] [apscheduler.scheduler] Added job "QueueCleanupWorker._cleanup_queue" to job store "default" gunicorn-web stdout | 2023-01-17 04:49:31,543 [70] [DEBUG] [util.ipresolver] Finished building AWS IP ranges queuecleanupworker stdout | 2023-01-17 04:49:31,545 [78] [DEBUG] [apscheduler.scheduler] Looking for jobs to run queuecleanupworker stdout | 2023-01-17 04:49:31,545 [78] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-18 00:07:43.535392+00:00 (in 69491.989911 seconds) notificationworker stdout | 2023-01-17 04:49:31,638 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:49:31,638 [76] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added notificationworker stdout | 2023-01-17 04:49:31,645 [76] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:35.646354 namespacegcworker stdout | 2023-01-17 04:49:31,940 [74] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" namespacegcworker stdout | 2023-01-17 04:49:31,940 [74] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:52:06.941185 namespacegcworker stdout | 2023-01-17 04:49:31,940 [74] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" namespacegcworker stdout | 2023-01-17 04:49:31,940 [74] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:42.941536 namespacegcworker stdout | 2023-01-17 04:49:31,940 [74] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" namespacegcworker stdout | 2023-01-17 04:49:31,942 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:49:31,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:41.148245+00:00 (in 9.205804 seconds) builder stdout | 2023-01-17 04:49:32,244 [60] [DEBUG] [__main__] Asking to start build manager with lifecycle "ephemeral" builder stdout | 2023-01-17 04:49:32,245 [60] [DEBUG] [__main__] Will pass buildman hostname quayregistry-quay-builder-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443 to builders for websocket connection builder stdout | 2023-01-17 04:49:32,245 [60] [DEBUG] [__main__] Starting build manager with lifecycle "ephemeral" builder stdout | 2023-01-17 04:49:32,246 [60] [DEBUG] [buildman.server] Initializing the lifecycle manager builder stdout | 2023-01-17 04:49:32,246 [60] [DEBUG] [buildman.manager.ephemeral] calling orchestrator_from_config builder stdout | 2023-01-17 04:49:32,246 [60] [DEBUG] [buildman.orchestrator] attempting to create orchestrator redis with kwargs {'host': 'quayregistry-quay-redis', 'password': '', 'skip_keyspace_event_setup': False, 'ssl': False, 'orchestrator_prefix': 'buildman/production/', 'canceller_only': False} builder stdout | 2023-01-17 04:49:32,246 [60] [DEBUG] [buildman.orchestrator] creating pubsub with key buildman/production/orchestrator_events gunicorn-web stdout | 2023-01-17 04:49:32,342 [70] [DEBUG] [data.database] Configuring database gunicorn-web stdout | 2023-01-17 04:49:32,344 [70] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:49:32,345 [70] [INFO] [data.secscan_model] =============================== gunicorn-web stdout | 2023-01-17 04:49:32,345 [70] [INFO] [data.secscan_model] Using split secscan model: `[]` gunicorn-web stdout | 2023-01-17 04:49:32,434 [70] [INFO] [data.secscan_model] =============================== gunicorn-web stdout | 2023-01-17 04:49:32,434 [70] [DEBUG] [data.logs_model] Configuring log lodel gunicorn-web stdout | 2023-01-17 04:49:32,434 [70] [INFO] [data.logs_model] =============================== gunicorn-web stdout | 2023-01-17 04:49:32,434 [70] [INFO] [data.logs_model] Using logs model `` gunicorn-web stdout | 2023-01-17 04:49:32,435 [70] [INFO] [data.logs_model] =============================== notificationworker stdout | 2023-01-17 04:49:32,634 [76] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" notificationworker stdout | 2023-01-17 04:49:32,635 [76] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:53:36.635997 notificationworker stdout | 2023-01-17 04:49:32,635 [76] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" notificationworker stdout | 2023-01-17 04:49:32,635 [76] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:01.636362 notificationworker stdout | 2023-01-17 04:49:32,635 [76] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" notificationworker stdout | 2023-01-17 04:49:32,645 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:49:32,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:35.646354+00:00 (in 2.911328 seconds) securityworker stdout | 2023-01-17 04:49:32,738 [87] [DEBUG] [workers.worker] Scheduling worker. securityworker stdout | 2023-01-17 04:49:32,739 [87] [INFO] [apscheduler.scheduler] Scheduler started builder stdout | 2023-01-17 04:49:32,845 [60] [DEBUG] [buildman.manager.ephemeral] setting on_key_change callbacks for job expiry, cancel builder stdout | 2023-01-17 04:49:32,845 [60] [DEBUG] [buildman.orchestrator] watching key: buildman/production/building/ builder stdout | 2023-01-17 04:49:32,845 [60] [DEBUG] [buildman.orchestrator] watching key: buildman/production/cancel/ builder stdout | 2023-01-17 04:49:32,845 [60] [DEBUG] [buildman.server] Initializing the gRPC server builder stdout | 2023-01-17 04:49:32,849 [60] [DEBUG] [buildman.server] Starting the gRPC server... securityworker stdout | 2023-01-17 04:49:32,934 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:49:32,934 [87] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added securityworker stdout | 2023-01-17 04:49:32,936 [87] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:46.936988 builder stdout | 2023-01-17 04:49:33,041 [60] [DEBUG] [buildman.server] Starting the build manager... builder stdout | 2023-01-17 04:49:33,041 [60] [DEBUG] [buildman.manager.ephemeral] Initializing work checker builder stdout | 2023-01-17 04:49:33,042 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-secscan stdout | 2023-01-17 04:49:33,240 [69] [INFO] [data.registry_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:49:33,240 [69] [INFO] [data.registry_model] Using registry model `` gunicorn-secscan stdout | 2023-01-17 04:49:33,241 [69] [INFO] [data.registry_model] =============================== builder stdout | 2023-01-17 04:49:33,339 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 04:49:33,446 [68] [INFO] [data.registry_model] =============================== gunicorn-registry stdout | 2023-01-17 04:49:33,446 [68] [INFO] [data.registry_model] Using registry model `` gunicorn-registry stdout | 2023-01-17 04:49:33,534 [68] [INFO] [data.registry_model] =============================== securityworker stdout | 2023-01-17 04:49:33,536 [87] [INFO] [apscheduler.scheduler] Added job "SecurityWorker._index_in_scanner" to job store "default" securityworker stdout | 2023-01-17 04:49:33,536 [87] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:49.537472 securityworker stdout | 2023-01-17 04:49:33,536 [87] [INFO] [apscheduler.scheduler] Added job "SecurityWorker._index_recent_manifests_in_scanner" to job store "default" securityworker stdout | 2023-01-17 04:49:33,544 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:49:33,544 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:46.936988+00:00 (in 13.392370 seconds) gunicorn-secscan stdout | 2023-01-17 04:49:34,238 [69] [DEBUG] [app] Loading default config. gunicorn-secscan stdout | 2023-01-17 04:49:34,238 [69] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml gunicorn-secscan stdout | 2023-01-17 04:49:34,347 [69] [DEBUG] [app] Loaded config gunicorn-registry stdout | 2023-01-17 04:49:34,446 [68] [DEBUG] [app] Loading default config. gunicorn-registry stdout | 2023-01-17 04:49:34,446 [68] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml gunicorn-secscan stdout | 2023-01-17 04:49:34,445 [69] [INFO] [util.ipresolver] Loading AWS IP ranges from disk gunicorn-secscan stdout | 2023-01-17 04:49:34,539 [69] [DEBUG] [util.ipresolver] Building AWS IP ranges expiredappspecifictokenworker stdout | 2023-01-17 04:49:34,636 [64] [DEBUG] [__main__] Starting expired app specific token GC worker gunicorn-registry stdout | 2023-01-17 04:49:34,639 [68] [DEBUG] [app] Loaded config gunicorn-registry stdout | 2023-01-17 04:49:34,639 [68] [INFO] [util.ipresolver] Loading AWS IP ranges from disk buildlogsarchiver stdout | 2023-01-17 04:49:34,644 [61] [DEBUG] [workers.worker] Scheduling worker. buildlogsarchiver stdout | 2023-01-17 04:49:34,646 [61] [INFO] [apscheduler.scheduler] Scheduler started gunicorn-registry stdout | 2023-01-17 04:49:34,736 [68] [DEBUG] [util.ipresolver] Building AWS IP ranges buildlogsarchiver stdout | 2023-01-17 04:49:34,737 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:49:34,737 [61] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:40.738568 expiredappspecifictokenworker stdout | 2023-01-17 04:49:34,740 [64] [DEBUG] [__main__] Found expiration window: 1d expiredappspecifictokenworker stdout | 2023-01-17 04:49:34,742 [64] [DEBUG] [workers.worker] Scheduling worker. expiredappspecifictokenworker stdout | 2023-01-17 04:49:34,742 [64] [INFO] [apscheduler.scheduler] Scheduler started expiredappspecifictokenworker stdout | 2023-01-17 04:49:34,743 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 04:49:34,743 [64] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added expiredappspecifictokenworker stdout | 2023-01-17 04:49:34,743 [64] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:21:32.744591 buildlogsarchiver stdout | 2023-01-17 04:49:34,738 [61] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added gunicorn-secscan stdout | 2023-01-17 04:49:35,144 [69] [DEBUG] [util.ipresolver] Finished building AWS IP ranges storagereplication stdout | 2023-01-17 04:49:35,243 [89] [DEBUG] [__main__] Full storage replication disabled; skipping expiredappspecifictokenworker stdout | 2023-01-17 04:49:35,338 [64] [INFO] [apscheduler.scheduler] Added job "ExpiredAppSpecificTokenWorker._gc_expired_tokens" to job store "default" expiredappspecifictokenworker stdout | 2023-01-17 04:49:35,338 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 04:49:35,338 [64] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:32.744591+00:00 (in 1917.405658 seconds) buildlogsarchiver stdout | 2023-01-17 04:49:35,339 [61] [INFO] [apscheduler.scheduler] Added job "ArchiveBuildLogsWorker._archive_redis_buildlogs" to job store "default" buildlogsarchiver stdout | 2023-01-17 04:49:35,340 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:49:35,340 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:40.738568+00:00 (in 5.398403 seconds) servicekey stdout | 2023-01-17 04:49:35,439 [88] [DEBUG] [workers.worker] Scheduling worker. servicekey stdout | 2023-01-17 04:49:35,440 [88] [INFO] [apscheduler.scheduler] Scheduler started servicekey stdout | 2023-01-17 04:49:35,440 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 04:49:35,440 [88] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added servicekey stdout | 2023-01-17 04:49:35,440 [88] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:15:30.441589 gunicorn-registry stdout | 2023-01-17 04:49:35,442 [68] [DEBUG] [util.ipresolver] Finished building AWS IP ranges notificationworker stdout | 2023-01-17 04:49:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:49:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:49:35 UTC)" (scheduled at 2023-01-17 04:49:35.646354+00:00) notificationworker stdout | 2023-01-17 04:49:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:45.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 04:49:35,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:49:35,737 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 49, 35, 736248), True, datetime.datetime(2023, 1, 17, 4, 49, 35, 736248), 0, 'notification/%', 50, 1, 0]) gunicorn-secscan stdout | 2023-01-17 04:49:35,738 [69] [DEBUG] [data.database] Configuring database gunicorn-registry stdout | 2023-01-17 04:49:35,739 [68] [DEBUG] [data.database] Configuring database gunicorn-secscan stdout | 2023-01-17 04:49:35,740 [69] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-secscan stdout | 2023-01-17 04:49:35,740 [69] [INFO] [data.secscan_model] =============================== gunicorn-registry stdout | 2023-01-17 04:49:35,741 [68] [INFO] [data.database] Connection pooling enabled for postgresql; stale timeout: None; max connection count: None gunicorn-secscan stdout | 2023-01-17 04:49:35,740 [69] [INFO] [data.secscan_model] Using split secscan model: `[]` gunicorn-secscan stdout | 2023-01-17 04:49:35,741 [69] [INFO] [data.secscan_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:49:35,741 [69] [DEBUG] [data.logs_model] Configuring log lodel gunicorn-secscan stdout | 2023-01-17 04:49:35,741 [69] [INFO] [data.logs_model] =============================== gunicorn-secscan stdout | 2023-01-17 04:49:35,741 [69] [INFO] [data.logs_model] Using logs model `` gunicorn-secscan stdout | 2023-01-17 04:49:35,741 [69] [INFO] [data.logs_model] =============================== gunicorn-registry stdout | 2023-01-17 04:49:35,741 [68] [INFO] [data.secscan_model] =============================== gunicorn-registry stdout | 2023-01-17 04:49:35,741 [68] [INFO] [data.secscan_model] Using split secscan model: `[]` gunicorn-registry stdout | 2023-01-17 04:49:35,742 [68] [INFO] [data.secscan_model] =============================== gunicorn-registry stdout | 2023-01-17 04:49:35,742 [68] [DEBUG] [data.logs_model] Configuring log lodel gunicorn-registry stdout | 2023-01-17 04:49:35,742 [68] [INFO] [data.logs_model] =============================== gunicorn-registry stdout | 2023-01-17 04:49:35,742 [68] [INFO] [data.logs_model] Using logs model `` gunicorn-registry stdout | 2023-01-17 04:49:35,742 [68] [INFO] [data.logs_model] =============================== repositorygcworker stdout | 2023-01-17 04:49:35,749 [80] [DEBUG] [__main__] Starting repository GC worker gunicorn-secscan stdout | 2023-01-17 04:49:35,843 [69] [DEBUG] [__config__] Starting secscan gunicorn with 2 workers and gevent worker class notificationworker stdout | 2023-01-17 04:49:35,843 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:49:35,844 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:49:35,844 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:49:45 UTC)" executed successfully servicekey stdout | 2023-01-17 04:49:35,844 [88] [INFO] [apscheduler.scheduler] Added job "ServiceKeyWorker._refresh_service_key" to job store "default" servicekey stdout | 2023-01-17 04:49:35,844 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 04:49:35,844 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:30.441589+00:00 (in 1554.596738 seconds) repositorygcworker stdout | 2023-01-17 04:49:35,934 [80] [DEBUG] [workers.worker] Scheduling worker. repositorygcworker stdout | 2023-01-17 04:49:35,935 [80] [INFO] [apscheduler.scheduler] Scheduler started repositorygcworker stdout | 2023-01-17 04:49:35,935 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:49:35,935 [80] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added repositorygcworker stdout | 2023-01-17 04:49:35,935 [80] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:04.936835 manifestbackfillworker stdout | 2023-01-17 04:49:35,949 [72] [DEBUG] [workers.worker] Scheduling worker. manifestbackfillworker stdout | 2023-01-17 04:49:35,950 [72] [INFO] [apscheduler.scheduler] Scheduler started manifestbackfillworker stdout | 2023-01-17 04:49:35,950 [72] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2023-01-17 04:49:35,950 [72] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added manifestbackfillworker stdout | 2023-01-17 04:49:35,950 [72] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:37:06.951421 chunkcleanupworker stdout | 2023-01-17 04:49:35,953 [62] [DEBUG] [__main__] Swift storage not detected; sleeping repositoryactioncounter stdout | 2023-01-17 04:49:36,240 [79] [DEBUG] [workers.worker] Scheduling worker. repositoryactioncounter stdout | 2023-01-17 04:49:36,240 [79] [INFO] [apscheduler.scheduler] Scheduler started repositoryactioncounter stdout | 2023-01-17 04:49:36,242 [79] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:49:36,242 [81] [DEBUG] [__main__] Starting security scanning notification worker repositoryactioncounter stdout | 2023-01-17 04:49:36,242 [79] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 08:35:53.243509 repositoryactioncounter stdout | 2023-01-17 04:49:36,242 [79] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added manifestbackfillworker stdout | 2023-01-17 04:49:36,245 [72] [INFO] [apscheduler.scheduler] Added job "ManifestBackfillWorker._backfill_manifests" to job store "default" manifestbackfillworker stdout | 2023-01-17 04:49:36,246 [72] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2023-01-17 04:49:36,246 [72] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:06.951421+00:00 (in 2850.705345 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:49:36,254 [81] [DEBUG] [workers.worker] Scheduling worker. securityscanningnotificationworker stdout | 2023-01-17 04:49:36,255 [81] [INFO] [apscheduler.scheduler] Scheduler started securityscanningnotificationworker stdout | 2023-01-17 04:49:36,255 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:49:36,255 [81] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added securityscanningnotificationworker stdout | 2023-01-17 04:49:36,255 [81] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:37.256652 globalpromstats stdout | 2023-01-17 04:49:36,342 [67] [DEBUG] [workers.worker] Scheduling worker. globalpromstats stdout | 2023-01-17 04:49:36,342 [67] [INFO] [apscheduler.scheduler] Scheduler started globalpromstats stdout | 2023-01-17 04:49:36,342 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 04:49:36,343 [67] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added globalpromstats stdout | 2023-01-17 04:49:36,343 [67] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:43:39.344159 repositorygcworker stdout | 2023-01-17 04:49:36,344 [80] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" repositorygcworker stdout | 2023-01-17 04:49:36,344 [80] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:51:17.345601 repositorygcworker stdout | 2023-01-17 04:49:36,344 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:49:36,344 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:04.936835+00:00 (in 28.591999 seconds) repositorygcworker stdout | 2023-01-17 04:49:36,345 [80] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" repositorygcworker stdout | 2023-01-17 04:49:36,345 [80] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:19.346120 repositorygcworker stdout | 2023-01-17 04:49:36,345 [80] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" repositorygcworker stdout | 2023-01-17 04:49:36,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:49:36,345 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:04.936835+00:00 (in 28.591297 seconds) repositorygcworker stdout | 2023-01-17 04:49:36,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:49:36,345 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:04.936835+00:00 (in 28.591215 seconds) repositoryactioncounter stdout | 2023-01-17 04:49:36,456 [79] [INFO] [apscheduler.scheduler] Added job "RepositoryActionCountWorker._run_counting" to job store "default" repositoryactioncounter stdout | 2023-01-17 04:49:36,456 [79] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositoryactioncounter stdout | 2023-01-17 04:49:36,456 [79] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 08:35:53.243509+00:00 (in 13576.787145 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:49:36,535 [81] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" securityscanningnotificationworker stdout | 2023-01-17 04:49:36,536 [81] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:51:23.537048 securityscanningnotificationworker stdout | 2023-01-17 04:49:36,536 [81] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" securityscanningnotificationworker stdout | 2023-01-17 04:49:36,536 [81] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:49:37.537407 securityscanningnotificationworker stdout | 2023-01-17 04:49:36,536 [81] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" securityscanningnotificationworker stdout | 2023-01-17 04:49:36,536 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:49:36,536 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:37.256652+00:00 (in 0.719815 seconds) teamsyncworker stdout | 2023-01-17 04:49:36,545 [90] [DEBUG] [__main__] Team syncing is disabled; sleeping globalpromstats stdout | 2023-01-17 04:49:36,550 [67] [INFO] [apscheduler.scheduler] Added job "GlobalPrometheusStatsWorker._try_report_stats" to job store "default" globalpromstats stdout | 2023-01-17 04:49:36,550 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 04:49:36,550 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:39.344159+00:00 (in 3242.793484 seconds) gunicorn-registry stdout | 2023-01-17 04:49:36,656 [68] [DEBUG] [__config__] Starting registry gunicorn with 8 workers and gevent worker class gunicorn-web stdout | 2023-01-17 04:49:36,681 [70] [DEBUG] [__config__] Starting web gunicorn with 4 workers and gevent worker class blobuploadcleanupworker stdout | 2023-01-17 04:49:36,760 [59] [DEBUG] [workers.worker] Scheduling worker. blobuploadcleanupworker stdout | 2023-01-17 04:49:36,761 [59] [INFO] [apscheduler.scheduler] Scheduler started blobuploadcleanupworker stdout | 2023-01-17 04:49:36,761 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2023-01-17 04:49:36,761 [59] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added blobuploadcleanupworker stdout | 2023-01-17 04:49:36,761 [59] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 05:34:07.762380 blobuploadcleanupworker stdout | 2023-01-17 04:49:36,812 [59] [INFO] [apscheduler.scheduler] Added job "BlobUploadCleanupWorker._try_cleanup_uploads" to job store "default" blobuploadcleanupworker stdout | 2023-01-17 04:49:36,812 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2023-01-17 04:49:36,812 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:07.762380+00:00 (in 2670.949764 seconds) exportactionlogsworker stdout | 2023-01-17 04:49:36,839 [65] [DEBUG] [__main__] Starting export action logs worker exportactionlogsworker stdout | 2023-01-17 04:49:36,851 [65] [DEBUG] [workers.worker] Scheduling worker. exportactionlogsworker stdout | 2023-01-17 04:49:36,851 [65] [INFO] [apscheduler.scheduler] Scheduler started exportactionlogsworker stdout | 2023-01-17 04:49:36,852 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:49:36,852 [65] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added exportactionlogsworker stdout | 2023-01-17 04:49:36,852 [65] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:35.853396 exportactionlogsworker stdout | 2023-01-17 04:49:36,962 [65] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default" exportactionlogsworker stdout | 2023-01-17 04:49:36,962 [65] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:51:54.963413 exportactionlogsworker stdout | 2023-01-17 04:49:36,962 [65] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default" exportactionlogsworker stdout | 2023-01-17 04:49:36,962 [65] [DEBUG] [workers.worker] First run scheduled for 2023-01-17 04:50:28.963662 exportactionlogsworker stdout | 2023-01-17 04:49:36,962 [65] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default" exportactionlogsworker stdout | 2023-01-17 04:49:36,962 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:49:36,962 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:28.963662+00:00 (in 52.000701 seconds) gunicorn-web stdout | 2023-01-17 04:49:37,061 [233] [DEBUG] [app] Starting request: urn:request:7edaed1f-3fb7-4d04-b9a8-a032cb2312df (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:49:37,067 [233] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-web stdout | 2023-01-17 04:49:37,082 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:49:37,084 [243] [DEBUG] [app] Starting request: urn:request:8c11e8e5-7146-43f4-81ab-92ea6c4ae939 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:49:37,086 [243] [DEBUG] [app] Ending request: urn:request:8c11e8e5-7146-43f4-81ab-92ea6c4ae939 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c11e8e5-7146-43f4-81ab-92ea6c4ae939', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 162 0.004) gunicorn-registry stdout | 2023-01-17 04:49:37,086 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:49:37,087 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:49:37,088 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:49:37,090 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:49:37,093 [231] [DEBUG] [app] Starting request: urn:request:0b91a9a7-20e7-4590-af8b-8a7e64eaa56d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:49:37,094 [231] [DEBUG] [app] Ending request: urn:request:0b91a9a7-20e7-4590-af8b-8a7e64eaa56d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b91a9a7-20e7-4590-af8b-8a7e64eaa56d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 159 0.004) gunicorn-web stdout | 2023-01-17 04:49:37,094 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:49:37,094 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:49:37,096 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 49, 37, 95489), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 49, 37, 95510), 'quay', None, datetime.datetime(2023, 1, 16, 4, 49, 37, 95523), datetime.datetime(2023, 1, 10, 4, 49, 37, 95534), None]) gunicorn-web stdout | 2023-01-17 04:49:37,108 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:49:37,114 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:49:37,118 [233] [DEBUG] [app] Ending request: urn:request:7edaed1f-3fb7-4d04-b9a8-a032cb2312df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7edaed1f-3fb7-4d04-b9a8-a032cb2312df', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:49:37,118 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:49:37,119 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:49:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:49:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.061 47 0.060) gunicorn-web stdout | 2023-01-17 04:49:37,121 [231] [DEBUG] [app] Starting request: urn:request:8b99d8f7-c626-47e8-a10a-50e11260cf3f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:49:37,126 [231] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-web stdout | 2023-01-17 04:49:37,143 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:49:37,145 [230] [DEBUG] [app] Starting request: urn:request:f0fa816f-ac18-4ab9-bf8c-010e1cf3d6be (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:49:37,147 [230] [DEBUG] [app] Ending request: urn:request:f0fa816f-ac18-4ab9-bf8c-010e1cf3d6be (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0fa816f-ac18-4ab9-bf8c-010e1cf3d6be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 162 0.004) gunicorn-registry stdout | 2023-01-17 04:49:37,147 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:49:37,147 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:49:37,149 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:49:37,151 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:49:37,153 [236] [DEBUG] [app] Starting request: urn:request:514b3ae0-688b-4ed1-aa52-415917e5ff7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:49:37,155 [236] [DEBUG] [app] Ending request: urn:request:514b3ae0-688b-4ed1-aa52-415917e5ff7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:514b3ae0-688b-4ed1-aa52-415917e5ff7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 159 0.004) gunicorn-web stdout | 2023-01-17 04:49:37,155 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:49:37,155 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:49:37,157 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 49, 37, 156703), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 49, 37, 156728), 'quay', None, datetime.datetime(2023, 1, 16, 4, 49, 37, 156744), datetime.datetime(2023, 1, 10, 4, 49, 37, 156757), None]) gunicorn-web stdout | 2023-01-17 04:49:37,169 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:49:37,176 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:49:37,179 [231] [DEBUG] [app] Ending request: urn:request:8b99d8f7-c626-47e8-a10a-50e11260cf3f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b99d8f7-c626-47e8-a10a-50e11260cf3f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:49:37,179 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:49:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.059 47 0.059) gunicorn-web stdout | 2023-01-17 04:49:37,180 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:49:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 04:49:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:49:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:49:37 UTC)" (scheduled at 2023-01-17 04:49:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:49:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:49:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 49, 37, 257546), True, datetime.datetime(2023, 1, 17, 4, 49, 37, 257546), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:49:37,258 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:37.537407+00:00 (in 0.280372 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:49:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:49:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:49:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:49:37,539 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:49:37,539 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:37.256652+00:00 (in 59.717107 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:49:37,539 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:37 UTC)" (scheduled at 2023-01-17 04:49:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:49:37,539 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:49:37,539 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:37 UTC)" executed successfully nginx stdout | 10.131.0.21 - - [17/Jan/2023:04:49:38 +0000] "GET / HTTP/1.1" 301 169 "-" "curl/7.61.1" buildlogsarchiver stdout | 2023-01-17 04:49:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:49:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:49:40 UTC)" (scheduled at 2023-01-17 04:49:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:49:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 49, 40, 739431), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:49:40,740 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:10.738568+00:00 (in 29.999698 seconds) buildlogsarchiver stdout | 2023-01-17 04:49:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:49:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:49:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:49:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:49:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:49:41 UTC)" (scheduled at 2023-01-17 04:49:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:49:41,148 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:49:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 49, 41, 148998), True, datetime.datetime(2023, 1, 17, 4, 49, 41, 148998), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:49:41,150 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:42.941536+00:00 (in 1.792976 seconds) namespacegcworker stdout | 2023-01-17 04:49:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:49:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:49:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:49:42,943 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:49:42,943 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:41.148245+00:00 (in 58.204707 seconds) namespacegcworker stdout | 2023-01-17 04:49:42,943 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:42 UTC)" (scheduled at 2023-01-17 04:49:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:49:42,943 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:49:42,944 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:42 UTC)" executed successfully builder stdout | 2023-01-17 04:49:43,349 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:49:43,362 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:49:43,362 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:49:43,372 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:49:45,735 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:49:45,735 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:55.646354+00:00 (in 9.910941 seconds) notificationworker stdout | 2023-01-17 04:49:45,735 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:49:55 UTC)" (scheduled at 2023-01-17 04:49:45.646354+00:00) notificationworker stdout | 2023-01-17 04:49:45,735 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:49:45,736 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 49, 45, 735847), True, datetime.datetime(2023, 1, 17, 4, 49, 45, 735847), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:49:45,747 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:49:45,747 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:49:45,747 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:49:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:49:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:49:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:49:46 UTC)" (scheduled at 2023-01-17 04:49:46.936988+00:00) securityworker stdout | 2023-01-17 04:49:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:49:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:49:46,938 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:49:49.537472+00:00 (in 2.600117 seconds) securityworker stdout | 2023-01-17 04:49:46,940 [87] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 securityworker stdout | 2023-01-17 04:49:46,957 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:49:46,959 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:49:46,968 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:49:46,971 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:49:46,972 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:49:49,538 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:49:49,539 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:16.936988+00:00 (in 27.398124 seconds) securityworker stdout | 2023-01-17 04:49:49,539 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:19 UTC)" (scheduled at 2023-01-17 04:49:49.537472+00:00) securityworker stdout | 2023-01-17 04:49:49,539 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:49:49,539 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:49:49,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:49:49,542 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:49:49,542 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:49:49,542 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:49:49,542 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:49:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:49:49,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:49:49,561 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:49:49,561 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:49:49,561 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:49:49,561 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:49:49,561 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:49:49,564 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:49:49,565 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:49:49,565 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:49:49,565 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:49:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:49:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:49:51 UTC)" (scheduled at 2023-01-17 04:49:51.348605+00:00) gcworker stdout | 2023-01-17 04:49:51,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 04:49:51,350 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:21.348605+00:00 (in 29.999653 seconds) gcworker stdout | 2023-01-17 04:49:51,361 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721391359, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:49:51,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:49:51,366 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:49:51,366 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:49:52,059 [236] [DEBUG] [app] Starting request: urn:request:8da9937e-70fd-4ce2-8fd6-648eb8f950cc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:49:52,061 [235] [DEBUG] [app] Starting request: urn:request:cf84bc78-f5ee-468c-8ef9-7c91439cbd9d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:49:52,065 [236] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-web stdout | 2023-01-17 04:49:52,066 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): localhost:8443 gunicorn-web stdout | 2023-01-17 04:49:52,079 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:49:52,080 [230] [DEBUG] [app] Starting request: urn:request:13b06628-b954-4bb1-839c-fb5ed7d7fab1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:49:52,081 [230] [DEBUG] [app] Ending request: urn:request:13b06628-b954-4bb1-839c-fb5ed7d7fab1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13b06628-b954-4bb1-839c-fb5ed7d7fab1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:49:52,081 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:49:52,082 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:49:52,082 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:49:52,083 [230] [DEBUG] [app] Starting request: urn:request:cc794936-b1c8-40a1-bc79-02465ab45169 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:49:52,083 [230] [DEBUG] [app] Ending request: urn:request:cc794936-b1c8-40a1-bc79-02465ab45169 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc794936-b1c8-40a1-bc79-02465ab45169', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:49:52,083 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:49:52,083 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:49:52,083 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:49:52,085 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:49:52,085 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:49:52,086 [231] [DEBUG] [app] Starting request: urn:request:f5d96c6b-acbb-4647-9290-678e561af946 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:49:52,087 [231] [DEBUG] [app] Ending request: urn:request:f5d96c6b-acbb-4647-9290-678e561af946 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f5d96c6b-acbb-4647-9290-678e561af946', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:49:52,087 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:49:52,087 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:49:52,087 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:49:52,088 [233] [DEBUG] [app] Starting request: urn:request:67f55b14-ec8d-4a1b-8e1f-ee8c7f37ff93 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:49:52,088 [233] [DEBUG] [app] Ending request: urn:request:67f55b14-ec8d-4a1b-8e1f-ee8c7f37ff93 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:67f55b14-ec8d-4a1b-8e1f-ee8c7f37ff93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:49:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:49:52,089 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:49:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:49:52,089 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:49:52,089 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 49, 52, 88599), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 49, 52, 88623), 'quay', None, datetime.datetime(2023, 1, 16, 4, 49, 52, 88639), datetime.datetime(2023, 1, 10, 4, 49, 52, 88655), None]) gunicorn-web stdout | 2023-01-17 04:49:52,091 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2023, 1, 17, 4, 49, 52, 90161), None, 'quay', 'quay', datetime.datetime(2023, 1, 17, 4, 49, 52, 90184), 'quay', None, datetime.datetime(2023, 1, 16, 4, 49, 52, 90196), datetime.datetime(2023, 1, 10, 4, 49, 52, 90209), None]) gunicorn-web stdout | 2023-01-17 04:49:52,102 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:49:52,102 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:49:52,108 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:49:52,108 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:49:52,111 [236] [DEBUG] [app] Ending request: urn:request:8da9937e-70fd-4ce2-8fd6-648eb8f950cc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8da9937e-70fd-4ce2-8fd6-648eb8f950cc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:49:52,112 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:49:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.054 47 0.053) gunicorn-web stdout | 2023-01-17 04:49:52,112 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:49:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:49:52,112 [235] [DEBUG] [app] Ending request: urn:request:cf84bc78-f5ee-468c-8ef9-7c91439cbd9d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf84bc78-f5ee-468c-8ef9-7c91439cbd9d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:49:52,112 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:49:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.053 47 0.054) gunicorn-web stdout | 2023-01-17 04:49:52,112 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:49:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:49:53,376 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:49:53,388 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:49:53,388 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:49:53,399 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:49:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:49:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:01.636362+00:00 (in 5.989512 seconds) notificationworker stdout | 2023-01-17 04:49:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:05 UTC)" (scheduled at 2023-01-17 04:49:55.646354+00:00) notificationworker stdout | 2023-01-17 04:49:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:49:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 49, 55, 647272), True, datetime.datetime(2023, 1, 17, 4, 49, 55, 647272), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:49:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:49:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:49:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:49:56,658 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:49:56,779 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:49:57,565 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:49:58,177 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:49:58,178 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:49:58,452 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:49:58,582 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:50:00,448 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:50:00,449 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:50:00,450 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:50:00,451 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:50:00,453 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:50:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:05.646354+00:00 (in 4.009458 seconds) notificationworker stdout | 2023-01-17 04:50:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:01 UTC)" (scheduled at 2023-01-17 04:50:01.636362+00:00) notificationworker stdout | 2023-01-17 04:50:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:50:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:50:01,679 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:50:02,053 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:50:03,142 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:50:03,170 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 04:50:03,409 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:03,422 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:03,422 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:03,433 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:50:03,463 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:50:03,974 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:50:04,070 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 04:50:04,370 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-secscan stdout | 2023-01-17 04:50:04,459 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:50:04,461 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:50:04,463 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositoryactioncounter stdout | 2023-01-17 04:50:04,549 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:50:04,653 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:50:04,655 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:50:04,657 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:50:04,659 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:50:04,661 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:50:04,663 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:50:04,665 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:50:04,667 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:50:04,668 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} globalpromstats stdout | 2023-01-17 04:50:04,875 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} repositorygcworker stdout | 2023-01-17 04:50:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:50:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:04 UTC)" (scheduled at 2023-01-17 04:50:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:50:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:50:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 4, 937652), True, datetime.datetime(2023, 1, 17, 4, 50, 4, 937652), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:50:04,939 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:19.346120+00:00 (in 14.408986 seconds) repositorygcworker stdout | 2023-01-17 04:50:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:50:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:50:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:04 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:50:05,381 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:50:05,550 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:50:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:15.646354+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 04:50:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:15 UTC)" (scheduled at 2023-01-17 04:50:05.646354+00:00) notificationworker stdout | 2023-01-17 04:50:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 5, 647190), True, datetime.datetime(2023, 1, 17, 4, 50, 5, 647190), 0, 'notification/%', 50, 1, 0]) blobuploadcleanupworker stdout | 2023-01-17 04:50:05,655 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 04:50:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:15 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:50:07,060 [231] [DEBUG] [app] Starting request: urn:request:f3b9bc7c-b5ca-4264-8686-98c7e596a459 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:07,060 [235] [DEBUG] [app] Starting request: urn:request:2aa3f71f-a4a4-46b2-b8fc-d57ecee0d741 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:07,065 [243] [DEBUG] [app] Starting request: urn:request:9b88036d-554a-42b1-a1ec-c28132562bdd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:07,065 [242] [DEBUG] [app] Starting request: urn:request:d9637fed-33f8-4d03-a948-5ef4c1a24c1c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:07,065 [243] [DEBUG] [app] Ending request: urn:request:9b88036d-554a-42b1-a1ec-c28132562bdd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b88036d-554a-42b1-a1ec-c28132562bdd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:50:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:50:07,066 [242] [DEBUG] [app] Ending request: urn:request:d9637fed-33f8-4d03-a948-5ef4c1a24c1c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d9637fed-33f8-4d03-a948-5ef4c1a24c1c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:50:07,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:07,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:07,070 [235] [DEBUG] [app] Starting request: urn:request:4aae2d4b-7898-4807-89ed-295fdfeff7ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:07,070 [235] [DEBUG] [app] Ending request: urn:request:4aae2d4b-7898-4807-89ed-295fdfeff7ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4aae2d4b-7898-4807-89ed-295fdfeff7ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:50:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:07,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:07,072 [236] [DEBUG] [app] Starting request: urn:request:7f1ba9c9-b6ec-4b19-a47a-90f147ae025d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:07,072 [236] [DEBUG] [app] Ending request: urn:request:7f1ba9c9-b6ec-4b19-a47a-90f147ae025d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f1ba9c9-b6ec-4b19-a47a-90f147ae025d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:07,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:50:07,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:07,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:07,086 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:07,090 [231] [DEBUG] [app] Ending request: urn:request:f3b9bc7c-b5ca-4264-8686-98c7e596a459 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f3b9bc7c-b5ca-4264-8686-98c7e596a459', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:07,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:07,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.032) gunicorn-web stdout | 2023-01-17 04:50:07,095 [235] [DEBUG] [app] Ending request: urn:request:2aa3f71f-a4a4-46b2-b8fc-d57ecee0d741 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2aa3f71f-a4a4-46b2-b8fc-d57ecee0d741', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:07,096 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:07,096 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.037) buildlogsarchiver stdout | 2023-01-17 04:50:10,740 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:50:10,740 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:40.738568+00:00 (in 29.998041 seconds) buildlogsarchiver stdout | 2023-01-17 04:50:10,740 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:40 UTC)" (scheduled at 2023-01-17 04:50:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:50:10,741 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 50, 10, 740894), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:50:10,752 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:50:10,752 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:50:10,752 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:40 UTC)" executed successfully builder stdout | 2023-01-17 04:50:13,434 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:13,447 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:13,447 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:13,458 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:25.646354+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 04:50:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:25 UTC)" (scheduled at 2023-01-17 04:50:15.646354+00:00) notificationworker stdout | 2023-01-17 04:50:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 15, 647224), True, datetime.datetime(2023, 1, 17, 4, 50, 15, 647224), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:15,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:15,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:15,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:50:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:19.537472+00:00 (in 2.600041 seconds) securityworker stdout | 2023-01-17 04:50:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:46 UTC)" (scheduled at 2023-01-17 04:50:16.936988+00:00) securityworker stdout | 2023-01-17 04:50:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:50:19,348 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:50:19,348 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:04.936835+00:00 (in 45.588406 seconds) repositorygcworker stdout | 2023-01-17 04:50:19,348 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:19 UTC)" (scheduled at 2023-01-17 04:50:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:50:19,348 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:50:19,348 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:50:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:46.936988+00:00 (in 27.399054 seconds) securityworker stdout | 2023-01-17 04:50:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:49 UTC)" (scheduled at 2023-01-17 04:50:19.537472+00:00) securityworker stdout | 2023-01-17 04:50:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:50:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:50:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:50:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:50:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:50:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:50:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:50:21,350 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:50:21,350 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:51.348605+00:00 (in 29.998459 seconds) gcworker stdout | 2023-01-17 04:50:21,350 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:51 UTC)" (scheduled at 2023-01-17 04:50:21.348605+00:00) gcworker stdout | 2023-01-17 04:50:21,351 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721421350, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:50:21,363 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:50:21,363 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:50:21,363 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:50:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:50:22,060 [236] [DEBUG] [app] Starting request: urn:request:0bb68531-d7c1-4af7-91d0-735d5178fa1d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:22,060 [231] [DEBUG] [app] Starting request: urn:request:56f0e37f-c846-4677-b488-507bc2c9aed6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:22,066 [230] [DEBUG] [app] Starting request: urn:request:1cde7665-42b1-44a8-8d1c-be15077694d3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:22,066 [243] [DEBUG] [app] Starting request: urn:request:8277bce1-ed32-47fd-931b-69038eafa0d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:22,066 [230] [DEBUG] [app] Ending request: urn:request:1cde7665-42b1-44a8-8d1c-be15077694d3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1cde7665-42b1-44a8-8d1c-be15077694d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:22,066 [243] [DEBUG] [app] Ending request: urn:request:8277bce1-ed32-47fd-931b-69038eafa0d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8277bce1-ed32-47fd-931b-69038eafa0d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:50:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:50:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:22,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:22,071 [235] [DEBUG] [app] Starting request: urn:request:b14a7fe2-3a95-41cb-9716-a6c382ef1e15 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:22,072 [233] [DEBUG] [app] Starting request: urn:request:73e0047c-4e0d-4f28-aaf3-cbbd2d130013 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:22,072 [235] [DEBUG] [app] Ending request: urn:request:b14a7fe2-3a95-41cb-9716-a6c382ef1e15 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b14a7fe2-3a95-41cb-9716-a6c382ef1e15', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:22,072 [233] [DEBUG] [app] Ending request: urn:request:73e0047c-4e0d-4f28-aaf3-cbbd2d130013 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73e0047c-4e0d-4f28-aaf3-cbbd2d130013', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:50:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:22,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:22,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:22,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:22,089 [231] [DEBUG] [app] Ending request: urn:request:56f0e37f-c846-4677-b488-507bc2c9aed6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:56f0e37f-c846-4677-b488-507bc2c9aed6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:22,089 [236] [DEBUG] [app] Ending request: urn:request:0bb68531-d7c1-4af7-91d0-735d5178fa1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0bb68531-d7c1-4af7-91d0-735d5178fa1d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:22,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:50:22,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) builder stdout | 2023-01-17 04:50:23,468 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:23,482 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:23,482 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:23,492 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:35.646354+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 04:50:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:35 UTC)" (scheduled at 2023-01-17 04:50:25.646354+00:00) notificationworker stdout | 2023-01-17 04:50:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 25, 647264), True, datetime.datetime(2023, 1, 17, 4, 50, 25, 647264), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:25,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:25,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:25,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:50:26,683 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:50:26,800 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:50:27,591 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 04:50:28,189 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 04:50:28,191 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 04:50:28,487 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:50:28,600 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:50:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:50:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:28 UTC)" (scheduled at 2023-01-17 04:50:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:50:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:50:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:50:28 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:50:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:35.853396+00:00 (in 6.889414 seconds) gunicorn-web stdout | 2023-01-17 04:50:30,455 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:50:30,457 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:50:30,460 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:50:30,462 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:50:30,465 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:50:31,708 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:50:32,089 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:50:33,150 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:50:33,205 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:50:33,470 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:50:33,502 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:33,515 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:33,515 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:33,525 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 04:50:34,002 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:50:34,105 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 04:50:34,405 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-secscan stdout | 2023-01-17 04:50:34,467 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:50:34,469 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:50:34,471 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositoryactioncounter stdout | 2023-01-17 04:50:34,581 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:50:34,660 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:50:34,665 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:50:34,667 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:50:34,669 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:50:34,672 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:50:34,674 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:50:34,677 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:50:34,679 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:50:34,681 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} globalpromstats stdout | 2023-01-17 04:50:34,886 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:50:35,412 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:50:35,558 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:50:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:45.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 04:50:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:45 UTC)" (scheduled at 2023-01-17 04:50:35.646354+00:00) notificationworker stdout | 2023-01-17 04:50:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 35, 647229), True, datetime.datetime(2023, 1, 17, 4, 50, 35, 647229), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:45 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 04:50:35,690 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} exportactionlogsworker stdout | 2023-01-17 04:50:35,854 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:50:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:28.963662+00:00 (in 53.109010 seconds) exportactionlogsworker stdout | 2023-01-17 04:50:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:35 UTC)" (scheduled at 2023-01-17 04:50:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:50:35,855 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:50:35,856 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 35, 855084), True, datetime.datetime(2023, 1, 17, 4, 50, 35, 855084), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:50:35,866 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:50:35,866 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:50:35,866 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:35 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:50:37,059 [233] [DEBUG] [app] Starting request: urn:request:c0bc8c4b-72a8-4a02-b578-4e4a1341f680 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:37,059 [231] [DEBUG] [app] Starting request: urn:request:653bab40-c9e2-4079-b106-060dcbc3840f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:37,064 [230] [DEBUG] [app] Starting request: urn:request:e1e5dbf3-3995-4e6d-b4e2-35400eaa1f97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:37,065 [230] [DEBUG] [app] Ending request: urn:request:e1e5dbf3-3995-4e6d-b4e2-35400eaa1f97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e1e5dbf3-3995-4e6d-b4e2-35400eaa1f97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:50:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:50:37,065 [240] [DEBUG] [app] Starting request: urn:request:a7ac6b4e-b135-4e72-a4a8-e0e777dc67ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:37,066 [240] [DEBUG] [app] Ending request: urn:request:a7ac6b4e-b135-4e72-a4a8-e0e777dc67ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a7ac6b4e-b135-4e72-a4a8-e0e777dc67ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.002) gunicorn-registry stdout | 2023-01-17 04:50:37,067 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:37,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:37,069 [235] [DEBUG] [app] Starting request: urn:request:ea302340-0270-4dbb-9479-f7cb3cd49509 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:37,070 [235] [DEBUG] [app] Ending request: urn:request:ea302340-0270-4dbb-9479-f7cb3cd49509 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ea302340-0270-4dbb-9479-f7cb3cd49509', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:50:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:37,071 [235] [DEBUG] [app] Starting request: urn:request:32fb27d1-fbbe-4d83-87df-0f3fcabe4c99 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:37,071 [235] [DEBUG] [app] Ending request: urn:request:32fb27d1-fbbe-4d83-87df-0f3fcabe4c99 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32fb27d1-fbbe-4d83-87df-0f3fcabe4c99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:37,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:50:37,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:37,087 [231] [DEBUG] [app] Ending request: urn:request:653bab40-c9e2-4079-b106-060dcbc3840f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:653bab40-c9e2-4079-b106-060dcbc3840f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:50:37,087 [233] [DEBUG] [app] Ending request: urn:request:c0bc8c4b-72a8-4a02-b578-4e4a1341f680 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0bc8c4b-72a8-4a02-b578-4e4a1341f680', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 04:50:37,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:37,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 04:50:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:50:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:37.537407+00:00 (in 0.280284 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:50:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:37 UTC)" (scheduled at 2023-01-17 04:50:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:50:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:50:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 37, 257553), True, datetime.datetime(2023, 1, 17, 4, 50, 37, 257553), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:50:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:50:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:50:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:50:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:50:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:23.537048+00:00 (in 45.999241 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:50:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:37 UTC)" (scheduled at 2023-01-17 04:50:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:50:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:50:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:50:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:50:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:10.738568+00:00 (in 29.999506 seconds) buildlogsarchiver stdout | 2023-01-17 04:50:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:10 UTC)" (scheduled at 2023-01-17 04:50:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:50:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 50, 40, 739465), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:50:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:50:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:50:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:50:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:50:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:42.941536+00:00 (in 1.792830 seconds) namespacegcworker stdout | 2023-01-17 04:50:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:41 UTC)" (scheduled at 2023-01-17 04:50:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:50:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:50:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 41, 149115), True, datetime.datetime(2023, 1, 17, 4, 50, 41, 149115), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:50:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:50:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:50:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:50:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:50:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:41.148245+00:00 (in 58.206232 seconds) namespacegcworker stdout | 2023-01-17 04:50:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:42 UTC)" (scheduled at 2023-01-17 04:50:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:50:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:50:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:51:42 UTC)" executed successfully builder stdout | 2023-01-17 04:50:43,535 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:43,548 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:43,548 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:43,558 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:55.646354+00:00 (in 9.999565 seconds) notificationworker stdout | 2023-01-17 04:50:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:55 UTC)" (scheduled at 2023-01-17 04:50:45.646354+00:00) notificationworker stdout | 2023-01-17 04:50:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 45, 647223), True, datetime.datetime(2023, 1, 17, 4, 50, 45, 647223), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:50:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:50:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:50:49.537472+00:00 (in 2.600017 seconds) securityworker stdout | 2023-01-17 04:50:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:16 UTC)" (scheduled at 2023-01-17 04:50:46.936988+00:00) securityworker stdout | 2023-01-17 04:50:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:50:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:50:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:16.936988+00:00 (in 27.399043 seconds) securityworker stdout | 2023-01-17 04:50:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:19 UTC)" (scheduled at 2023-01-17 04:50:49.537472+00:00) securityworker stdout | 2023-01-17 04:50:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:50:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:50:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:50:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:50:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:50:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:50:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:50:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:50:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:50:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:50:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:50:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:50:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:50:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:50:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:50:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:21.348605+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 04:50:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:21 UTC)" (scheduled at 2023-01-17 04:50:51.348605+00:00) gcworker stdout | 2023-01-17 04:50:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721451349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:50:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:50:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:50:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:50:52,060 [233] [DEBUG] [app] Starting request: urn:request:4afc1390-ffa4-4ef6-a9de-18897de9770f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:52,060 [235] [DEBUG] [app] Starting request: urn:request:8551fcef-5835-40f2-bab4-588d4b112745 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:50:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:52,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:50:52,065 [243] [DEBUG] [app] Starting request: urn:request:21d6085d-9c89-4273-8a20-4f09538bdb23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:52,066 [242] [DEBUG] [app] Starting request: urn:request:0a59d7d3-40ce-4fe1-84dd-b8583ad73604 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:50:52,066 [243] [DEBUG] [app] Ending request: urn:request:21d6085d-9c89-4273-8a20-4f09538bdb23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:21d6085d-9c89-4273-8a20-4f09538bdb23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:50:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.003) gunicorn-registry stdout | 2023-01-17 04:50:52,066 [242] [DEBUG] [app] Ending request: urn:request:0a59d7d3-40ce-4fe1-84dd-b8583ad73604 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a59d7d3-40ce-4fe1-84dd-b8583ad73604', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:50:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:50:52,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:50:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:50:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:50:52,071 [231] [DEBUG] [app] Starting request: urn:request:6dc4c718-f434-4f10-a5aa-25e636112918 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:52,071 [231] [DEBUG] [app] Ending request: urn:request:6dc4c718-f434-4f10-a5aa-25e636112918 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6dc4c718-f434-4f10-a5aa-25e636112918', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:52,071 [236] [DEBUG] [app] Starting request: urn:request:98ac80c2-e745-4a37-8713-b4e79c82124c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:50:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:50:52,072 [236] [DEBUG] [app] Ending request: urn:request:98ac80c2-e745-4a37-8713-b4e79c82124c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98ac80c2-e745-4a37-8713-b4e79c82124c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:50:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:50:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:50:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:50:52,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:50:52,088 [235] [DEBUG] [app] Ending request: urn:request:8551fcef-5835-40f2-bab4-588d4b112745 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8551fcef-5835-40f2-bab4-588d4b112745', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:52,088 [233] [DEBUG] [app] Ending request: urn:request:4afc1390-ffa4-4ef6-a9de-18897de9770f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4afc1390-ffa4-4ef6-a9de-18897de9770f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:50:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:50:52,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 04:50:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:50:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:50:53,568 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:50:53,580 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:50:53,580 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:50:53,591 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:50:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:01.636362+00:00 (in 5.989526 seconds) notificationworker stdout | 2023-01-17 04:50:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:05 UTC)" (scheduled at 2023-01-17 04:50:55.646354+00:00) notificationworker stdout | 2023-01-17 04:50:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:50:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 50, 55, 647238), True, datetime.datetime(2023, 1, 17, 4, 50, 55, 647238), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:50:55,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:50:55,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:50:55,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:50:56,718 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:50:56,835 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:50:57,622 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:50:58,198 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:50:58,224 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:50:58,497 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:50:58,634 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:51:00,463 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:51:00,465 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:51:00,468 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:51:00,470 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:51:00,473 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:51:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:05.646354+00:00 (in 4.009512 seconds) notificationworker stdout | 2023-01-17 04:51:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:01 UTC)" (scheduled at 2023-01-17 04:51:01.636362+00:00) notificationworker stdout | 2023-01-17 04:51:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:51:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:51:01,720 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:51:02,100 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:51:03,186 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:51:03,226 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:51:03,506 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:51:03,601 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:03,614 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:03,614 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:03,625 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 04:51:04,009 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:51:04,140 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 04:51:04,441 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-secscan stdout | 2023-01-17 04:51:04,474 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:51:04,476 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:51:04,479 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositoryactioncounter stdout | 2023-01-17 04:51:04,590 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:51:04,668 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:51:04,673 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:51:04,677 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:51:04,680 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:51:04,682 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:51:04,684 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:51:04,686 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:51:04,689 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:51:04,691 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} globalpromstats stdout | 2023-01-17 04:51:04,921 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} repositorygcworker stdout | 2023-01-17 04:51:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:51:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:17.345601+00:00 (in 12.408260 seconds) repositorygcworker stdout | 2023-01-17 04:51:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:04 UTC)" (scheduled at 2023-01-17 04:51:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:51:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:51:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 4, 937786), True, datetime.datetime(2023, 1, 17, 4, 51, 4, 937786), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:51:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:51:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:51:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:04 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:51:05,442 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:51:05,592 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:51:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:15.646354+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 04:51:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:15 UTC)" (scheduled at 2023-01-17 04:51:05.646354+00:00) notificationworker stdout | 2023-01-17 04:51:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 5, 647258), True, datetime.datetime(2023, 1, 17, 4, 51, 5, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:15 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 04:51:05,700 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:51:07,060 [236] [DEBUG] [app] Starting request: urn:request:7af88cdc-2d9c-44ad-bda3-90bc09e88836 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:07,060 [231] [DEBUG] [app] Starting request: urn:request:5dd1524b-8341-4991-bb39-d76520f18ec8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:07,065 [230] [DEBUG] [app] Starting request: urn:request:a70bef7f-2f26-4301-8514-40036d207dde (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,065 [242] [DEBUG] [app] Starting request: urn:request:5a247b25-942a-4863-a171-191524f299d0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,065 [230] [DEBUG] [app] Ending request: urn:request:a70bef7f-2f26-4301-8514-40036d207dde (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a70bef7f-2f26-4301-8514-40036d207dde', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,065 [242] [DEBUG] [app] Ending request: urn:request:5a247b25-942a-4863-a171-191524f299d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a247b25-942a-4863-a171-191524f299d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:51:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:51:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:07,070 [235] [DEBUG] [app] Starting request: urn:request:822ea8e2-c5f9-4004-beb4-363ef32ab7ac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:07,071 [235] [DEBUG] [app] Ending request: urn:request:822ea8e2-c5f9-4004-beb4-363ef32ab7ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:822ea8e2-c5f9-4004-beb4-363ef32ab7ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:07,071 [233] [DEBUG] [app] Starting request: urn:request:54e55acf-2316-4912-99ab-15d19939e187 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:51:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:07,071 [233] [DEBUG] [app] Ending request: urn:request:54e55acf-2316-4912-99ab-15d19939e187 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:54e55acf-2316-4912-99ab-15d19939e187', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:07,088 [231] [DEBUG] [app] Ending request: urn:request:5dd1524b-8341-4991-bb39-d76520f18ec8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5dd1524b-8341-4991-bb39-d76520f18ec8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:07,088 [236] [DEBUG] [app] Ending request: urn:request:7af88cdc-2d9c-44ad-bda3-90bc09e88836 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7af88cdc-2d9c-44ad-bda3-90bc09e88836', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:51:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) buildlogsarchiver stdout | 2023-01-17 04:51:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:51:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:40.738568+00:00 (in 29.999505 seconds) buildlogsarchiver stdout | 2023-01-17 04:51:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:40 UTC)" (scheduled at 2023-01-17 04:51:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:51:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 51, 10, 739441), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:51:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:51:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:51:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:40 UTC)" executed successfully builder stdout | 2023-01-17 04:51:13,627 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:13,639 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:13,639 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:13,650 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:25.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 04:51:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:25 UTC)" (scheduled at 2023-01-17 04:51:15.646354+00:00) notificationworker stdout | 2023-01-17 04:51:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 15, 647245), True, datetime.datetime(2023, 1, 17, 4, 51, 15, 647245), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:19.537472+00:00 (in 2.600004 seconds) securityworker stdout | 2023-01-17 04:51:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:46 UTC)" (scheduled at 2023-01-17 04:51:16.936988+00:00) securityworker stdout | 2023-01-17 04:51:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:51:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:51:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:19.346120+00:00 (in 2.000016 seconds) repositorygcworker stdout | 2023-01-17 04:51:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:17 UTC)" (scheduled at 2023-01-17 04:51:17.345601+00:00) repositorygcworker stdout | 2023-01-17 04:51:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 51, 17, 346558), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:51:17,358 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 17, 346558), True, datetime.datetime(2023, 1, 17, 4, 51, 17, 346558), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:51:17,362 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 17, 346558), True, datetime.datetime(2023, 1, 17, 4, 51, 17, 346558), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 4, 51, 17, 346558), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:51:17,365 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:51:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:51:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:51:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:04.936835+00:00 (in 45.590180 seconds) repositorygcworker stdout | 2023-01-17 04:51:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:19 UTC)" (scheduled at 2023-01-17 04:51:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:51:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:51:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:46.936988+00:00 (in 27.399013 seconds) securityworker stdout | 2023-01-17 04:51:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:49 UTC)" (scheduled at 2023-01-17 04:51:19.537472+00:00) securityworker stdout | 2023-01-17 04:51:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:51:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:51:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:51:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:51:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:51:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:51:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:51:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:51:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:51.348605+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 04:51:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:51 UTC)" (scheduled at 2023-01-17 04:51:21.348605+00:00) gcworker stdout | 2023-01-17 04:51:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721481349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:51:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:51:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:51:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:51:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:51:22,060 [236] [DEBUG] [app] Starting request: urn:request:fb89dd13-bfcc-49c6-bcfd-c0c17f499562 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:22,060 [231] [DEBUG] [app] Starting request: urn:request:35e90c5b-05f5-481b-817f-de897c4f4db7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:22,065 [230] [DEBUG] [app] Starting request: urn:request:a4725e6e-43ea-4e28-9bd7-3e673ec2a81d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:22,066 [230] [DEBUG] [app] Ending request: urn:request:a4725e6e-43ea-4e28-9bd7-3e673ec2a81d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4725e6e-43ea-4e28-9bd7-3e673ec2a81d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:22,065 [243] [DEBUG] [app] Starting request: urn:request:d2501ce1-9fda-4852-b9b4-08aeac6c9679 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:51:22,066 [243] [DEBUG] [app] Ending request: urn:request:d2501ce1-9fda-4852-b9b4-08aeac6c9679 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2501ce1-9fda-4852-b9b4-08aeac6c9679', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:51:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:51:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:22,071 [235] [DEBUG] [app] Starting request: urn:request:50fba349-826c-4488-8a4b-24ddf92ff3ce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:22,071 [236] [DEBUG] [app] Starting request: urn:request:d80a8d5a-225c-40e7-b743-72adf157e5ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:22,071 [235] [DEBUG] [app] Ending request: urn:request:50fba349-826c-4488-8a4b-24ddf92ff3ce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50fba349-826c-4488-8a4b-24ddf92ff3ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:22,071 [236] [DEBUG] [app] Ending request: urn:request:d80a8d5a-225c-40e7-b743-72adf157e5ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d80a8d5a-225c-40e7-b743-72adf157e5ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:22,088 [231] [DEBUG] [app] Ending request: urn:request:35e90c5b-05f5-481b-817f-de897c4f4db7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:35e90c5b-05f5-481b-817f-de897c4f4db7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:22,088 [236] [DEBUG] [app] Ending request: urn:request:fb89dd13-bfcc-49c6-bcfd-c0c17f499562 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb89dd13-bfcc-49c6-bcfd-c0c17f499562', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:51:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 04:51:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:51:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:37.256652+00:00 (in 13.719207 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:51:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:23 UTC)" (scheduled at 2023-01-17 04:51:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:51:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 51, 23, 537782), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:51:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 23, 537782), True, datetime.datetime(2023, 1, 17, 4, 51, 23, 537782), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:51:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 23, 537782), True, datetime.datetime(2023, 1, 17, 4, 51, 23, 537782), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 4, 51, 23, 537782), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:51:23,555 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:51:23,555 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:23 UTC)" executed successfully builder stdout | 2023-01-17 04:51:23,651 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:23,662 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:23,662 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:23,673 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:35.646354+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 04:51:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:35 UTC)" (scheduled at 2023-01-17 04:51:25.646354+00:00) notificationworker stdout | 2023-01-17 04:51:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 25, 647296), True, datetime.datetime(2023, 1, 17, 4, 51, 25, 647296), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:51:26,730 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:51:26,869 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:51:27,632 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:51:28,234 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:51:28,253 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:51:28,532 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:51:28,672 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:51:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:51:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:35.853396+00:00 (in 6.889289 seconds) exportactionlogsworker stdout | 2023-01-17 04:51:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:28 UTC)" (scheduled at 2023-01-17 04:51:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:51:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:51:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:28 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:51:30,470 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:51:30,472 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:51:30,475 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:51:30,478 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:51:30,480 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:51:31,731 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:51:32,123 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:51:33,221 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:51:33,241 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:51:33,537 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:51:33,681 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:33,694 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:33,694 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:33,704 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 04:51:34,038 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:51:34,175 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 04:51:34,476 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-secscan stdout | 2023-01-17 04:51:34,482 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:51:34,485 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:51:34,487 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositoryactioncounter stdout | 2023-01-17 04:51:34,605 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:51:34,675 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:51:34,679 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:51:34,685 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:51:34,687 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:51:34,690 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:51:34,692 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:51:34,695 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:51:34,697 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:51:34,699 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} globalpromstats stdout | 2023-01-17 04:51:34,957 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:51:35,448 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:51:35,616 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:51:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:45.646354+00:00 (in 9.999491 seconds) notificationworker stdout | 2023-01-17 04:51:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:45 UTC)" (scheduled at 2023-01-17 04:51:35.646354+00:00) notificationworker stdout | 2023-01-17 04:51:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 35, 647262), True, datetime.datetime(2023, 1, 17, 4, 51, 35, 647262), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:45 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 04:51:35,736 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} exportactionlogsworker stdout | 2023-01-17 04:51:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:51:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:54.963413+00:00 (in 19.109505 seconds) exportactionlogsworker stdout | 2023-01-17 04:51:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:35 UTC)" (scheduled at 2023-01-17 04:51:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:51:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:51:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 35, 854257), True, datetime.datetime(2023, 1, 17, 4, 51, 35, 854257), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:51:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:51:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:51:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:35 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:51:37,059 [231] [DEBUG] [app] Starting request: urn:request:bf04aa02-bd09-4d99-8fbe-38b5ab224514 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:37,059 [236] [DEBUG] [app] Starting request: urn:request:8248e60d-5ee0-4346-8816-6e803f3556d4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:37,060 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:37,064 [230] [DEBUG] [app] Starting request: urn:request:619477a4-6ab8-46f8-9d1c-2b9a769d694e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:37,064 [243] [DEBUG] [app] Starting request: urn:request:af49b2d2-298e-4edd-9f50-ee09dddc426a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:37,064 [230] [DEBUG] [app] Ending request: urn:request:619477a4-6ab8-46f8-9d1c-2b9a769d694e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:619477a4-6ab8-46f8-9d1c-2b9a769d694e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:51:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:51:37,065 [243] [DEBUG] [app] Ending request: urn:request:af49b2d2-298e-4edd-9f50-ee09dddc426a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af49b2d2-298e-4edd-9f50-ee09dddc426a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:51:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:51:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:37,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:37,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:37,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:37,069 [235] [DEBUG] [app] Starting request: urn:request:f94a7bb6-cb5e-44a2-84d0-f4bf3b2fd5f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:37,069 [233] [DEBUG] [app] Starting request: urn:request:31b2e8d2-f031-4f1c-ad93-5392440f9b59 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:37,069 [235] [DEBUG] [app] Ending request: urn:request:f94a7bb6-cb5e-44a2-84d0-f4bf3b2fd5f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f94a7bb6-cb5e-44a2-84d0-f4bf3b2fd5f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:37,070 [233] [DEBUG] [app] Ending request: urn:request:31b2e8d2-f031-4f1c-ad93-5392440f9b59 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31b2e8d2-f031-4f1c-ad93-5392440f9b59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:37,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:37,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:37,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:37,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:37,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:37,086 [231] [DEBUG] [app] Ending request: urn:request:bf04aa02-bd09-4d99-8fbe-38b5ab224514 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf04aa02-bd09-4d99-8fbe-38b5ab224514', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:37,086 [236] [DEBUG] [app] Ending request: urn:request:8248e60d-5ee0-4346-8816-6e803f3556d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8248e60d-5ee0-4346-8816-6e803f3556d4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:37,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:37,086 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:37,086 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:51:37,086 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) securityscanningnotificationworker stdout | 2023-01-17 04:51:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:51:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:37.537407+00:00 (in 0.280329 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:51:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:37 UTC)" (scheduled at 2023-01-17 04:51:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:51:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:51:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 37, 257476), True, datetime.datetime(2023, 1, 17, 4, 51, 37, 257476), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:51:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:51:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:51:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:51:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:51:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:37.256652+00:00 (in 59.718841 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:51:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:37 UTC)" (scheduled at 2023-01-17 04:51:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:51:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:51:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:51:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:51:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:10.738568+00:00 (in 29.999553 seconds) buildlogsarchiver stdout | 2023-01-17 04:51:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:10 UTC)" (scheduled at 2023-01-17 04:51:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:51:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 51, 40, 739409), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:51:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:51:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:51:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:51:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:51:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:42.941536+00:00 (in 1.792836 seconds) namespacegcworker stdout | 2023-01-17 04:51:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:41 UTC)" (scheduled at 2023-01-17 04:51:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:51:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:51:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 41, 149082), True, datetime.datetime(2023, 1, 17, 4, 51, 41, 149082), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:51:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:51:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:51:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:51:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:51:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:06.941185+00:00 (in 23.999190 seconds) namespacegcworker stdout | 2023-01-17 04:51:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:42 UTC)" (scheduled at 2023-01-17 04:51:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:51:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:51:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:52:42 UTC)" executed successfully builder stdout | 2023-01-17 04:51:43,715 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:43,727 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:43,727 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:43,738 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:55.646354+00:00 (in 9.999504 seconds) notificationworker stdout | 2023-01-17 04:51:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:55 UTC)" (scheduled at 2023-01-17 04:51:45.646354+00:00) notificationworker stdout | 2023-01-17 04:51:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 45, 647263), True, datetime.datetime(2023, 1, 17, 4, 51, 45, 647263), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:51:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:51:49.537472+00:00 (in 2.599982 seconds) securityworker stdout | 2023-01-17 04:51:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:16 UTC)" (scheduled at 2023-01-17 04:51:46.936988+00:00) securityworker stdout | 2023-01-17 04:51:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:51:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:51:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:16.936988+00:00 (in 27.399010 seconds) securityworker stdout | 2023-01-17 04:51:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:19 UTC)" (scheduled at 2023-01-17 04:51:49.537472+00:00) securityworker stdout | 2023-01-17 04:51:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:51:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:51:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:51:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:51:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:51:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:51:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:51:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:51:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:51:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:51:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:51:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:51:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:51:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:51:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:51:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:21.348605+00:00 (in 29.999506 seconds) gcworker stdout | 2023-01-17 04:51:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:21 UTC)" (scheduled at 2023-01-17 04:51:51.348605+00:00) gcworker stdout | 2023-01-17 04:51:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721511349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:51:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:51:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:51:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:51:52,060 [231] [DEBUG] [app] Starting request: urn:request:876db1a7-4c42-434b-a553-8fbfcb348ad0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:52,061 [236] [DEBUG] [app] Starting request: urn:request:a30029ba-cd6e-400e-9ab7-1069b4976097 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:51:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:51:52,065 [230] [DEBUG] [app] Starting request: urn:request:644b4f43-d0f7-4a77-867c-228b235734f5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:52,066 [242] [DEBUG] [app] Starting request: urn:request:e7e5c821-b287-4c93-9d78-e85b3fcecae2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:51:52,066 [230] [DEBUG] [app] Ending request: urn:request:644b4f43-d0f7-4a77-867c-228b235734f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:644b4f43-d0f7-4a77-867c-228b235734f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:51:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:51:52,066 [242] [DEBUG] [app] Ending request: urn:request:e7e5c821-b287-4c93-9d78-e85b3fcecae2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e7e5c821-b287-4c93-9d78-e85b3fcecae2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:51:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:51:52,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:51:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:51:52,070 [235] [DEBUG] [app] Starting request: urn:request:d2fc49c6-3e8a-459a-8cf7-233d85a98ce1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:52,071 [233] [DEBUG] [app] Starting request: urn:request:30945158-6d39-417c-8316-ff1f56e768c5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:51:52,071 [235] [DEBUG] [app] Ending request: urn:request:d2fc49c6-3e8a-459a-8cf7-233d85a98ce1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d2fc49c6-3e8a-459a-8cf7-233d85a98ce1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:51:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:52,071 [233] [DEBUG] [app] Ending request: urn:request:30945158-6d39-417c-8316-ff1f56e768c5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:30945158-6d39-417c-8316-ff1f56e768c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:51:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:51:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:51:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:51:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:51:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:51:52,088 [236] [DEBUG] [app] Ending request: urn:request:a30029ba-cd6e-400e-9ab7-1069b4976097 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a30029ba-cd6e-400e-9ab7-1069b4976097', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:51:52,088 [231] [DEBUG] [app] Ending request: urn:request:876db1a7-4c42-434b-a553-8fbfcb348ad0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:876db1a7-4c42-434b-a553-8fbfcb348ad0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:51:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:51:52,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 04:51:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:51:53,748 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:51:53,760 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:51:53,760 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:51:53,770 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:51:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:51:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:28.963662+00:00 (in 33.999796 seconds) exportactionlogsworker stdout | 2023-01-17 04:51:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:54 UTC)" (scheduled at 2023-01-17 04:51:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 04:51:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 51, 54, 964291), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:51:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 54, 964291), True, datetime.datetime(2023, 1, 17, 4, 51, 54, 964291), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:51:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 51, 54, 964291), True, datetime.datetime(2023, 1, 17, 4, 51, 54, 964291), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 4, 51, 54, 964291), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:51:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:51:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:56:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:51:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:51:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:01.636362+00:00 (in 5.989520 seconds) notificationworker stdout | 2023-01-17 04:51:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:05 UTC)" (scheduled at 2023-01-17 04:51:55.646354+00:00) notificationworker stdout | 2023-01-17 04:51:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:51:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 51, 55, 647255), True, datetime.datetime(2023, 1, 17, 4, 51, 55, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:51:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:51:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:51:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:51:56,761 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:51:56,904 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:51:57,654 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:51:58,270 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:51:58,288 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:51:58,568 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:51:58,707 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:52:00,479 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:52:00,481 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:52:00,483 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:52:00,485 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:52:00,487 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:52:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:05.646354+00:00 (in 4.009519 seconds) notificationworker stdout | 2023-01-17 04:52:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:01 UTC)" (scheduled at 2023-01-17 04:52:01.636362+00:00) notificationworker stdout | 2023-01-17 04:52:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:52:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:52:01,761 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:52:02,130 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:52:03,257 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:52:03,276 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:52:03,573 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:52:03,781 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:03,793 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:03,793 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:03,804 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 04:52:04,074 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:52:04,210 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:52:04,489 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:52:04,492 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:52:04,494 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:52:04,510 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 04:52:04,639 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:52:04,682 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:52:04,686 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:52:04,693 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:52:04,695 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:52:04,698 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:52:04,700 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:52:04,702 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:52:04,705 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 04:52:04,707 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} repositorygcworker stdout | 2023-01-17 04:52:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:52:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:19.346120+00:00 (in 14.408859 seconds) repositorygcworker stdout | 2023-01-17 04:52:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:04 UTC)" (scheduled at 2023-01-17 04:52:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:52:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:52:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 4, 937685), True, datetime.datetime(2023, 1, 17, 4, 52, 4, 937685), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:52:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:52:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:52:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 04:52:04,966 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:52:05,462 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 04:52:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:15.646354+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 04:52:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:15 UTC)" (scheduled at 2023-01-17 04:52:05.646354+00:00) notificationworker stdout | 2023-01-17 04:52:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 5, 647241), True, datetime.datetime(2023, 1, 17, 4, 52, 5, 647241), 0, 'notification/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:52:05,650 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 04:52:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:15 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 04:52:05,771 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} namespacegcworker stdout | 2023-01-17 04:52:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:52:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:41.148245+00:00 (in 34.206610 seconds) namespacegcworker stdout | 2023-01-17 04:52:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:57:06 UTC)" (scheduled at 2023-01-17 04:52:06.941185+00:00) namespacegcworker stdout | 2023-01-17 04:52:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 52, 6, 942031), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:52:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 52, 6, 942031), True, datetime.datetime(2023, 1, 17, 4, 52, 6, 942031), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:52:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 52, 6, 942031), True, datetime.datetime(2023, 1, 17, 4, 52, 6, 942031), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 4, 52, 6, 942031), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:52:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:52:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:57:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:52:07,060 [236] [DEBUG] [app] Starting request: urn:request:3bc9b4da-93ca-47b2-84d2-bd7c3a8d07de (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:07,060 [233] [DEBUG] [app] Starting request: urn:request:af20155e-6699-4b2f-b5e9-8b761c0fdd51 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:07,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:07,065 [230] [DEBUG] [app] Starting request: urn:request:3d7a5e1c-adef-4c83-8db8-161e899dfe64 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:07,065 [242] [DEBUG] [app] Starting request: urn:request:66bcd2a4-cbe9-495a-b696-482812aad978 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:07,065 [230] [DEBUG] [app] Ending request: urn:request:3d7a5e1c-adef-4c83-8db8-161e899dfe64 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d7a5e1c-adef-4c83-8db8-161e899dfe64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:52:07,066 [242] [DEBUG] [app] Ending request: urn:request:66bcd2a4-cbe9-495a-b696-482812aad978 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:66bcd2a4-cbe9-495a-b696-482812aad978', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:52:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:52:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:07,070 [235] [DEBUG] [app] Starting request: urn:request:39f2496f-26b6-4c98-989a-5f34d6c7d3f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:07,070 [235] [DEBUG] [app] Ending request: urn:request:39f2496f-26b6-4c98-989a-5f34d6c7d3f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39f2496f-26b6-4c98-989a-5f34d6c7d3f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:07,070 [231] [DEBUG] [app] Starting request: urn:request:7e45364c-6217-42ae-af2f-32b4bb733f45 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:07,071 [231] [DEBUG] [app] Ending request: urn:request:7e45364c-6217-42ae-af2f-32b4bb733f45 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e45364c-6217-42ae-af2f-32b4bb733f45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:52:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:07,087 [236] [DEBUG] [app] Ending request: urn:request:3bc9b4da-93ca-47b2-84d2-bd7c3a8d07de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3bc9b4da-93ca-47b2-84d2-bd7c3a8d07de', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 04:52:07,097 [233] [DEBUG] [app] Ending request: urn:request:af20155e-6699-4b2f-b5e9-8b761c0fdd51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af20155e-6699-4b2f-b5e9-8b761c0fdd51', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:07,097 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:07,098 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.039 47 0.038) buildlogsarchiver stdout | 2023-01-17 04:52:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:52:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:40.738568+00:00 (in 29.999514 seconds) buildlogsarchiver stdout | 2023-01-17 04:52:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:40 UTC)" (scheduled at 2023-01-17 04:52:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:52:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 52, 10, 739513), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:52:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:52:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:52:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:40 UTC)" executed successfully builder stdout | 2023-01-17 04:52:13,814 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:13,827 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:13,827 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:13,837 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:25.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 04:52:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:25 UTC)" (scheduled at 2023-01-17 04:52:15.646354+00:00) notificationworker stdout | 2023-01-17 04:52:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 15, 647265), True, datetime.datetime(2023, 1, 17, 4, 52, 15, 647265), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:19.537472+00:00 (in 2.599979 seconds) securityworker stdout | 2023-01-17 04:52:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:46 UTC)" (scheduled at 2023-01-17 04:52:16.936988+00:00) securityworker stdout | 2023-01-17 04:52:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:52:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:52:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:04.936835+00:00 (in 45.590269 seconds) repositorygcworker stdout | 2023-01-17 04:52:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:19 UTC)" (scheduled at 2023-01-17 04:52:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:52:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:52:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:46.936988+00:00 (in 27.399038 seconds) securityworker stdout | 2023-01-17 04:52:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:49 UTC)" (scheduled at 2023-01-17 04:52:19.537472+00:00) securityworker stdout | 2023-01-17 04:52:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:52:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:52:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:52:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:52:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:52:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:52:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:52:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:52:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:51.348605+00:00 (in 29.999554 seconds) gcworker stdout | 2023-01-17 04:52:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:51 UTC)" (scheduled at 2023-01-17 04:52:21.348605+00:00) gcworker stdout | 2023-01-17 04:52:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721541349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:52:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:52:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:52:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:52:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:52:22,059 [236] [DEBUG] [app] Starting request: urn:request:c7d6f02e-f2e0-4223-82c2-b20403b63712 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:22,059 [231] [DEBUG] [app] Starting request: urn:request:6851b296-8a98-4028-8e21-7fbdaeb8f406 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:22,064 [242] [DEBUG] [app] Starting request: urn:request:964a853b-19db-4fe7-9c6a-80ce9833bda7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:22,064 [242] [DEBUG] [app] Ending request: urn:request:964a853b-19db-4fe7-9c6a-80ce9833bda7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:964a853b-19db-4fe7-9c6a-80ce9833bda7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:22,064 [230] [DEBUG] [app] Starting request: urn:request:bbbce8d5-ddf6-42f7-b3f8-21ce2fdfca90 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:52:22,065 [230] [DEBUG] [app] Ending request: urn:request:bbbce8d5-ddf6-42f7-b3f8-21ce2fdfca90 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bbbce8d5-ddf6-42f7-b3f8-21ce2fdfca90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:52:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:52:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:22,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:22,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:22,069 [235] [DEBUG] [app] Starting request: urn:request:c826771c-d3cd-470e-9586-688a9da213ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:22,069 [233] [DEBUG] [app] Starting request: urn:request:3d12a236-046c-4e19-b72f-c35eca05cca1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:22,069 [235] [DEBUG] [app] Ending request: urn:request:c826771c-d3cd-470e-9586-688a9da213ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c826771c-d3cd-470e-9586-688a9da213ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:22,070 [233] [DEBUG] [app] Ending request: urn:request:3d12a236-046c-4e19-b72f-c35eca05cca1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d12a236-046c-4e19-b72f-c35eca05cca1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:22,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:52:22,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:22,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:22,087 [231] [DEBUG] [app] Ending request: urn:request:6851b296-8a98-4028-8e21-7fbdaeb8f406 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6851b296-8a98-4028-8e21-7fbdaeb8f406', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:52:22,087 [236] [DEBUG] [app] Ending request: urn:request:c7d6f02e-f2e0-4223-82c2-b20403b63712 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c7d6f02e-f2e0-4223-82c2-b20403b63712', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 04:52:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 04:52:23,846 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:23,859 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:23,859 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:23,869 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:35.646354+00:00 (in 9.999510 seconds) notificationworker stdout | 2023-01-17 04:52:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:35 UTC)" (scheduled at 2023-01-17 04:52:25.646354+00:00) notificationworker stdout | 2023-01-17 04:52:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 25, 647276), True, datetime.datetime(2023, 1, 17, 4, 52, 25, 647276), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:52:26,796 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:52:26,910 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:52:27,689 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:52:28,283 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:52:28,323 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:52:28,604 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:52:28,742 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:52:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:52:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:35.853396+00:00 (in 6.889195 seconds) exportactionlogsworker stdout | 2023-01-17 04:52:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:28 UTC)" (scheduled at 2023-01-17 04:52:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:52:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:52:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:28 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:52:30,486 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:52:30,488 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:52:30,491 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:52:30,493 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:52:30,495 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:52:31,767 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:52:32,166 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:52:33,283 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:52:33,293 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:52:33,591 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:52:33,875 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:33,887 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:33,887 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:33,898 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 04:52:34,101 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:52:34,229 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:52:34,497 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:52:34,499 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:52:34,502 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:52:34,545 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 04:52:34,654 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:52:34,687 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:52:34,692 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:52:34,700 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:52:34,703 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:52:34,705 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:52:34,707 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:52:34,713 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:52:34,715 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:52:34,717 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} globalpromstats stdout | 2023-01-17 04:52:34,997 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:52:35,498 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 04:52:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:45.646354+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 04:52:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:45 UTC)" (scheduled at 2023-01-17 04:52:35.646354+00:00) notificationworker stdout | 2023-01-17 04:52:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 35, 647240), True, datetime.datetime(2023, 1, 17, 4, 52, 35, 647240), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:52:35,683 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:52:35,806 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} exportactionlogsworker stdout | 2023-01-17 04:52:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:52:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:28.963662+00:00 (in 53.109848 seconds) exportactionlogsworker stdout | 2023-01-17 04:52:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:35 UTC)" (scheduled at 2023-01-17 04:52:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:52:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:52:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 35, 854167), True, datetime.datetime(2023, 1, 17, 4, 52, 35, 854167), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:52:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:52:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:52:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:35 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:52:37,060 [233] [DEBUG] [app] Starting request: urn:request:988b169e-7de1-4e6c-a0f4-c4b1aac10e5d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:37,060 [236] [DEBUG] [app] Starting request: urn:request:ec5c2d49-42bd-4620-9964-2a8de268de81 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:37,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:37,065 [230] [DEBUG] [app] Starting request: urn:request:bdcbf5b8-09bc-47f1-af4d-46dbf42d7e33 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:37,065 [230] [DEBUG] [app] Ending request: urn:request:bdcbf5b8-09bc-47f1-af4d-46dbf42d7e33 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bdcbf5b8-09bc-47f1-af4d-46dbf42d7e33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:52:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:52:37,066 [241] [DEBUG] [app] Starting request: urn:request:931a076f-1691-48b5-8967-eba1bb8c2765 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:37,067 [241] [DEBUG] [app] Ending request: urn:request:931a076f-1691-48b5-8967-eba1bb8c2765 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:931a076f-1691-48b5-8967-eba1bb8c2765', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:52:37,067 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:37,069 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:37,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:37,070 [235] [DEBUG] [app] Starting request: urn:request:47b7ea8c-1103-45ca-aed2-2ffef0b32eea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:37,070 [235] [DEBUG] [app] Ending request: urn:request:47b7ea8c-1103-45ca-aed2-2ffef0b32eea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47b7ea8c-1103-45ca-aed2-2ffef0b32eea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:37,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:37,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:37,071 [235] [DEBUG] [app] Starting request: urn:request:25d0fb9f-afea-4428-a68a-5ea136bc4c11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:37,071 [235] [DEBUG] [app] Ending request: urn:request:25d0fb9f-afea-4428-a68a-5ea136bc4c11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25d0fb9f-afea-4428-a68a-5ea136bc4c11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:37,087 [233] [DEBUG] [app] Ending request: urn:request:988b169e-7de1-4e6c-a0f4-c4b1aac10e5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:988b169e-7de1-4e6c-a0f4-c4b1aac10e5d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:37,087 [236] [DEBUG] [app] Ending request: urn:request:ec5c2d49-42bd-4620-9964-2a8de268de81 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec5c2d49-42bd-4620-9964-2a8de268de81', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:37,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:37,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:52:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) securityscanningnotificationworker stdout | 2023-01-17 04:52:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:52:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:37.537407+00:00 (in 0.280329 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:52:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:37 UTC)" (scheduled at 2023-01-17 04:52:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:52:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:52:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 37, 257557), True, datetime.datetime(2023, 1, 17, 4, 52, 37, 257557), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:52:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:52:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:52:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:52:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:52:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:37.256652+00:00 (in 59.718901 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:52:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:37 UTC)" (scheduled at 2023-01-17 04:52:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:52:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:52:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:52:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:52:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:10.738568+00:00 (in 29.999494 seconds) buildlogsarchiver stdout | 2023-01-17 04:52:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:10 UTC)" (scheduled at 2023-01-17 04:52:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:52:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 52, 40, 739465), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:52:40,757 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:52:40,757 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:52:40,757 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:52:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:52:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:42.941536+00:00 (in 1.792832 seconds) namespacegcworker stdout | 2023-01-17 04:52:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:41 UTC)" (scheduled at 2023-01-17 04:52:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:52:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:52:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 41, 149083), True, datetime.datetime(2023, 1, 17, 4, 52, 41, 149083), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:52:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:52:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:52:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:52:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:52:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:41.148245+00:00 (in 58.206205 seconds) namespacegcworker stdout | 2023-01-17 04:52:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:42 UTC)" (scheduled at 2023-01-17 04:52:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:52:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:52:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:53:42 UTC)" executed successfully builder stdout | 2023-01-17 04:52:43,908 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:43,921 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:43,921 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:43,932 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:55.646354+00:00 (in 9.999501 seconds) notificationworker stdout | 2023-01-17 04:52:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:55 UTC)" (scheduled at 2023-01-17 04:52:45.646354+00:00) notificationworker stdout | 2023-01-17 04:52:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 45, 647288), True, datetime.datetime(2023, 1, 17, 4, 52, 45, 647288), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:52:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:52:49.537472+00:00 (in 2.600037 seconds) securityworker stdout | 2023-01-17 04:52:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:16 UTC)" (scheduled at 2023-01-17 04:52:46.936988+00:00) securityworker stdout | 2023-01-17 04:52:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:46,956 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:46,959 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:46,959 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:52:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:52:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:16.936988+00:00 (in 27.399028 seconds) securityworker stdout | 2023-01-17 04:52:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:19 UTC)" (scheduled at 2023-01-17 04:52:49.537472+00:00) securityworker stdout | 2023-01-17 04:52:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:52:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:52:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:52:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:52:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:52:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:52:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:52:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:52:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:52:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:52:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:52:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:52:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:52:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:52:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:52:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:21.348605+00:00 (in 29.999572 seconds) gcworker stdout | 2023-01-17 04:52:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:21 UTC)" (scheduled at 2023-01-17 04:52:51.348605+00:00) gcworker stdout | 2023-01-17 04:52:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721571349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:52:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:52:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:52:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:52:52,060 [231] [DEBUG] [app] Starting request: urn:request:67a492e5-8387-4d7d-9365-189f52e007ae (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:52,060 [235] [DEBUG] [app] Starting request: urn:request:529d86ff-55f2-4a47-a620-d04deb3d3e0a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:52:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:52:52,065 [241] [DEBUG] [app] Starting request: urn:request:d76feeee-b897-46b7-a657-5fc4baac7b51 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:52,065 [240] [DEBUG] [app] Starting request: urn:request:fbff4b5a-b202-447d-ac7a-872c8f79f013 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:52:52,065 [241] [DEBUG] [app] Ending request: urn:request:d76feeee-b897-46b7-a657-5fc4baac7b51 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d76feeee-b897-46b7-a657-5fc4baac7b51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:52,066 [240] [DEBUG] [app] Ending request: urn:request:fbff4b5a-b202-447d-ac7a-872c8f79f013 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fbff4b5a-b202-447d-ac7a-872c8f79f013', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:52:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:52:52,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:52:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:52:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:52:52,070 [236] [DEBUG] [app] Starting request: urn:request:a4dc74c4-d8be-4e68-92fd-055bdbc0f1f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:52,071 [233] [DEBUG] [app] Starting request: urn:request:e4451f47-8533-4d50-be5d-a3c1eb318a71 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:52:52,071 [236] [DEBUG] [app] Ending request: urn:request:a4dc74c4-d8be-4e68-92fd-055bdbc0f1f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4dc74c4-d8be-4e68-92fd-055bdbc0f1f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:52:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:52:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:52,071 [233] [DEBUG] [app] Ending request: urn:request:e4451f47-8533-4d50-be5d-a3c1eb318a71 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4451f47-8533-4d50-be5d-a3c1eb318a71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:52:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:52:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:52:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:52:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:52:52,090 [235] [DEBUG] [app] Ending request: urn:request:529d86ff-55f2-4a47-a620-d04deb3d3e0a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:529d86ff-55f2-4a47-a620-d04deb3d3e0a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:52,090 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:52,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 04:52:52,096 [231] [DEBUG] [app] Ending request: urn:request:67a492e5-8387-4d7d-9365-189f52e007ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:67a492e5-8387-4d7d-9365-189f52e007ae', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:52:52,096 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:52:52,097 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.038 47 0.038) builder stdout | 2023-01-17 04:52:53,942 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:52:53,954 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:52:53,954 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:52:53,965 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:52:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:01.636362+00:00 (in 5.989521 seconds) notificationworker stdout | 2023-01-17 04:52:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:05 UTC)" (scheduled at 2023-01-17 04:52:55.646354+00:00) notificationworker stdout | 2023-01-17 04:52:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:52:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 52, 55, 647279), True, datetime.datetime(2023, 1, 17, 4, 52, 55, 647279), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:52:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:52:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:52:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:52:56,803 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:52:56,925 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:52:57,726 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:52:58,299 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:52:58,340 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:52:58,623 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:52:58,778 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:53:00,493 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:53:00,495 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:53:00,498 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:53:00,500 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:53:00,502 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:53:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:05.646354+00:00 (in 4.009542 seconds) notificationworker stdout | 2023-01-17 04:53:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:01 UTC)" (scheduled at 2023-01-17 04:53:01.636362+00:00) notificationworker stdout | 2023-01-17 04:53:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:53:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:53:01,800 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:53:02,172 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:53:03,319 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:53:03,326 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:53:03,603 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:53:03,975 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:03,988 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:03,988 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:03,998 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 04:53:04,132 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:53:04,264 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:53:04,504 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:53:04,507 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:53:04,509 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:53:04,573 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 04:53:04,689 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:53:04,699 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:53:04,701 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:53:04,707 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:53:04,711 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:53:04,713 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:53:04,718 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:53:04,723 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:53:04,725 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:53:04,727 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 04:53:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:53:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:19.346120+00:00 (in 14.408820 seconds) repositorygcworker stdout | 2023-01-17 04:53:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:04 UTC)" (scheduled at 2023-01-17 04:53:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:53:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:53:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 4, 937729), True, datetime.datetime(2023, 1, 17, 4, 53, 4, 937729), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:53:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:53:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:53:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 04:53:05,033 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:53:05,534 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 04:53:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:15.646354+00:00 (in 9.999544 seconds) notificationworker stdout | 2023-01-17 04:53:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:15 UTC)" (scheduled at 2023-01-17 04:53:05.646354+00:00) notificationworker stdout | 2023-01-17 04:53:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 5, 647281), True, datetime.datetime(2023, 1, 17, 4, 53, 5, 647281), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:15 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:53:05,719 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:53:05,841 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:53:07,060 [231] [DEBUG] [app] Starting request: urn:request:5c58dc0b-c4fd-4c0c-a847-2ee688db74c7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:07,060 [235] [DEBUG] [app] Starting request: urn:request:f2b8f52d-50ac-42f1-a0bd-394a72b1ce12 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:07,064 [241] [DEBUG] [app] Starting request: urn:request:d91f34ec-5683-454c-ba86-2eb45291e683 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:07,065 [242] [DEBUG] [app] Starting request: urn:request:39a79116-de46-465d-a6fb-a097a59452fe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:07,065 [241] [DEBUG] [app] Ending request: urn:request:d91f34ec-5683-454c-ba86-2eb45291e683 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d91f34ec-5683-454c-ba86-2eb45291e683', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:07,065 [242] [DEBUG] [app] Ending request: urn:request:39a79116-de46-465d-a6fb-a097a59452fe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:39a79116-de46-465d-a6fb-a097a59452fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:07,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:53:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:53:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:07,070 [236] [DEBUG] [app] Starting request: urn:request:336ad9aa-8396-45b5-a89a-3b622785e3bb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:07,070 [231] [DEBUG] [app] Starting request: urn:request:7832f1a8-5cae-4997-bb4b-21eb2930c5af (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:07,071 [236] [DEBUG] [app] Ending request: urn:request:336ad9aa-8396-45b5-a89a-3b622785e3bb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:336ad9aa-8396-45b5-a89a-3b622785e3bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:07,071 [231] [DEBUG] [app] Ending request: urn:request:7832f1a8-5cae-4997-bb4b-21eb2930c5af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7832f1a8-5cae-4997-bb4b-21eb2930c5af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:53:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:07,087 [235] [DEBUG] [app] Ending request: urn:request:f2b8f52d-50ac-42f1-a0bd-394a72b1ce12 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f2b8f52d-50ac-42f1-a0bd-394a72b1ce12', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:07,087 [231] [DEBUG] [app] Ending request: urn:request:5c58dc0b-c4fd-4c0c-a847-2ee688db74c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5c58dc0b-c4fd-4c0c-a847-2ee688db74c7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:53:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) buildlogsarchiver stdout | 2023-01-17 04:53:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:53:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:40.738568+00:00 (in 29.999520 seconds) buildlogsarchiver stdout | 2023-01-17 04:53:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:40 UTC)" (scheduled at 2023-01-17 04:53:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:53:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 53, 10, 739449), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:53:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:53:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:53:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:40 UTC)" executed successfully builder stdout | 2023-01-17 04:53:14,009 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:14,021 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:14,021 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:14,032 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:25.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 04:53:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:25 UTC)" (scheduled at 2023-01-17 04:53:15.646354+00:00) notificationworker stdout | 2023-01-17 04:53:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 15, 647280), True, datetime.datetime(2023, 1, 17, 4, 53, 15, 647280), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:19.537472+00:00 (in 2.599988 seconds) securityworker stdout | 2023-01-17 04:53:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:46 UTC)" (scheduled at 2023-01-17 04:53:16.936988+00:00) securityworker stdout | 2023-01-17 04:53:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:53:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:53:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:04.936835+00:00 (in 45.590237 seconds) repositorygcworker stdout | 2023-01-17 04:53:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:19 UTC)" (scheduled at 2023-01-17 04:53:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:53:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:53:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:46.936988+00:00 (in 27.399017 seconds) securityworker stdout | 2023-01-17 04:53:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:49 UTC)" (scheduled at 2023-01-17 04:53:19.537472+00:00) securityworker stdout | 2023-01-17 04:53:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:53:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:53:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:53:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:53:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:53:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:53:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:53:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:53:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:51.348605+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 04:53:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:51 UTC)" (scheduled at 2023-01-17 04:53:21.348605+00:00) gcworker stdout | 2023-01-17 04:53:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721601349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:53:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:53:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:53:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:53:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:53:22,060 [236] [DEBUG] [app] Starting request: urn:request:d31c693a-151e-425e-a055-b60bbd66b4ba (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:22,061 [231] [DEBUG] [app] Starting request: urn:request:20d35fdc-4b49-4aa1-8f1a-ca8174056765 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:22,065 [230] [DEBUG] [app] Starting request: urn:request:9b90d290-fe8c-4821-af38-0296212f8b2f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:22,066 [243] [DEBUG] [app] Starting request: urn:request:edf2df89-2023-4c1e-8dc8-2bf812206c44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:22,066 [230] [DEBUG] [app] Ending request: urn:request:9b90d290-fe8c-4821-af38-0296212f8b2f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b90d290-fe8c-4821-af38-0296212f8b2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:22,066 [243] [DEBUG] [app] Ending request: urn:request:edf2df89-2023-4c1e-8dc8-2bf812206c44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:edf2df89-2023-4c1e-8dc8-2bf812206c44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:53:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:53:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:22,071 [235] [DEBUG] [app] Starting request: urn:request:fdcd90cc-1eaa-4c3a-b9c1-d3b20b2088cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,071 [236] [DEBUG] [app] Starting request: urn:request:d8d7079e-b579-4953-a06e-21658cbe859e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:22,071 [235] [DEBUG] [app] Ending request: urn:request:fdcd90cc-1eaa-4c3a-b9c1-d3b20b2088cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdcd90cc-1eaa-4c3a-b9c1-d3b20b2088cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:22,071 [236] [DEBUG] [app] Ending request: urn:request:d8d7079e-b579-4953-a06e-21658cbe859e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8d7079e-b579-4953-a06e-21658cbe859e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:53:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:22,088 [236] [DEBUG] [app] Ending request: urn:request:d31c693a-151e-425e-a055-b60bbd66b4ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d31c693a-151e-425e-a055-b60bbd66b4ba', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:22,089 [231] [DEBUG] [app] Ending request: urn:request:20d35fdc-4b49-4aa1-8f1a-ca8174056765 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20d35fdc-4b49-4aa1-8f1a-ca8174056765', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 04:53:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:53:24,042 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:24,054 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:24,054 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:24,065 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:35.646354+00:00 (in 9.999501 seconds) notificationworker stdout | 2023-01-17 04:53:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:35 UTC)" (scheduled at 2023-01-17 04:53:25.646354+00:00) notificationworker stdout | 2023-01-17 04:53:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 25, 647338), True, datetime.datetime(2023, 1, 17, 4, 53, 25, 647338), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:25,662 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:25,662 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:25,662 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:53:26,839 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:53:26,929 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:53:27,761 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:53:28,311 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:53:28,375 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:53:28,659 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:53:28,806 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:53:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:53:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:35.853396+00:00 (in 6.889319 seconds) exportactionlogsworker stdout | 2023-01-17 04:53:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:28 UTC)" (scheduled at 2023-01-17 04:53:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:53:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:53:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:28 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:53:30,500 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:53:30,503 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:53:30,506 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:53:30,508 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:53:30,510 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:53:31,836 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:53:32,205 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:53:33,354 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:53:33,361 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:53:33,639 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:53:34,075 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:34,088 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:34,088 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:34,099 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 04:53:34,157 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:53:34,270 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:53:34,513 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:53:34,516 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:53:34,518 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:53:34,577 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 04:53:34,708 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:53:34,711 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:53:34,713 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:53:34,715 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:53:34,722 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:53:34,725 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:53:34,727 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:53:34,732 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:53:34,735 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:53:34,737 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} globalpromstats stdout | 2023-01-17 04:53:35,065 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:53:35,568 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 04:53:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:36.635997+00:00 (in 0.989159 seconds) notificationworker stdout | 2023-01-17 04:53:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:45 UTC)" (scheduled at 2023-01-17 04:53:35.646354+00:00) notificationworker stdout | 2023-01-17 04:53:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 35, 647300), True, datetime.datetime(2023, 1, 17, 4, 53, 35, 647300), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:53:35,754 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:53:35,846 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} exportactionlogsworker stdout | 2023-01-17 04:53:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:53:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:28.963662+00:00 (in 53.109910 seconds) exportactionlogsworker stdout | 2023-01-17 04:53:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:35 UTC)" (scheduled at 2023-01-17 04:53:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:53:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:53:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 35, 854136), True, datetime.datetime(2023, 1, 17, 4, 53, 35, 854136), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:53:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:53:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:53:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:53:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:45.646354+00:00 (in 9.009841 seconds) notificationworker stdout | 2023-01-17 04:53:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:58:36 UTC)" (scheduled at 2023-01-17 04:53:36.635997+00:00) notificationworker stdout | 2023-01-17 04:53:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 53, 36, 636910), 'notification/%']) notificationworker stdout | 2023-01-17 04:53:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 53, 36, 636910), True, datetime.datetime(2023, 1, 17, 4, 53, 36, 636910), 0, 'notification/%']) notificationworker stdout | 2023-01-17 04:53:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 53, 36, 636910), True, datetime.datetime(2023, 1, 17, 4, 53, 36, 636910), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 4, 53, 36, 636910), 'notification/%']) notificationworker stdout | 2023-01-17 04:53:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 04:58:36 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:53:37,060 [235] [DEBUG] [app] Starting request: urn:request:1ef2635a-f863-47ce-a96a-8f0fbac23ae3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:37,060 [233] [DEBUG] [app] Starting request: urn:request:7ff54fc5-a5e6-4e1a-b5a2-145612f15649 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:37,065 [243] [DEBUG] [app] Starting request: urn:request:16aa990e-d745-4c79-b33c-7765b2f77034 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:37,065 [239] [DEBUG] [app] Starting request: urn:request:4c38f0a5-a779-4488-b1f0-212bf3d29434 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:37,065 [243] [DEBUG] [app] Ending request: urn:request:16aa990e-d745-4c79-b33c-7765b2f77034 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:16aa990e-d745-4c79-b33c-7765b2f77034', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:53:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:53:37,066 [239] [DEBUG] [app] Ending request: urn:request:4c38f0a5-a779-4488-b1f0-212bf3d29434 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c38f0a5-a779-4488-b1f0-212bf3d29434', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:53:37,067 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:37,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:37,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:37,071 [231] [DEBUG] [app] Starting request: urn:request:4d81d9c9-223c-4c4a-aa1e-a953f3174e12 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:37,072 [231] [DEBUG] [app] Ending request: urn:request:4d81d9c9-223c-4c4a-aa1e-a953f3174e12 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d81d9c9-223c-4c4a-aa1e-a953f3174e12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:37,072 [236] [DEBUG] [app] Starting request: urn:request:25c53581-3e30-4ff0-afa4-65aa42d2520a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:37,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:53:37,072 [236] [DEBUG] [app] Ending request: urn:request:25c53581-3e30-4ff0-afa4-65aa42d2520a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25c53581-3e30-4ff0-afa4-65aa42d2520a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:37,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:37,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:37,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:53:37,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:37,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:37,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:37,089 [233] [DEBUG] [app] Ending request: urn:request:7ff54fc5-a5e6-4e1a-b5a2-145612f15649 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7ff54fc5-a5e6-4e1a-b5a2-145612f15649', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:37,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:37,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:53:37,089 [235] [DEBUG] [app] Ending request: urn:request:1ef2635a-f863-47ce-a96a-8f0fbac23ae3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ef2635a-f863-47ce-a96a-8f0fbac23ae3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:37,089 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 04:53:37,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 04:53:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:53:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:37.537407+00:00 (in 0.280349 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:53:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:37 UTC)" (scheduled at 2023-01-17 04:53:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:53:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:53:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 37, 257483), True, datetime.datetime(2023, 1, 17, 4, 53, 37, 257483), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:53:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:53:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:53:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:53:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:53:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:37.256652+00:00 (in 59.718860 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:53:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:37 UTC)" (scheduled at 2023-01-17 04:53:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:53:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:53:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:53:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:53:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:10.738568+00:00 (in 29.999507 seconds) buildlogsarchiver stdout | 2023-01-17 04:53:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:10 UTC)" (scheduled at 2023-01-17 04:53:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:53:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 53, 40, 739463), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:53:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:53:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:53:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:53:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:53:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:42.941536+00:00 (in 1.792824 seconds) namespacegcworker stdout | 2023-01-17 04:53:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:41 UTC)" (scheduled at 2023-01-17 04:53:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:53:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:53:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 41, 149117), True, datetime.datetime(2023, 1, 17, 4, 53, 41, 149117), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:53:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:53:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:53:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:53:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:53:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:41.148245+00:00 (in 58.206222 seconds) namespacegcworker stdout | 2023-01-17 04:53:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:42 UTC)" (scheduled at 2023-01-17 04:53:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:53:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:53:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:54:42 UTC)" executed successfully builder stdout | 2023-01-17 04:53:44,109 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:44,121 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:44,121 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:44,132 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:55.646354+00:00 (in 9.999496 seconds) notificationworker stdout | 2023-01-17 04:53:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:55 UTC)" (scheduled at 2023-01-17 04:53:45.646354+00:00) notificationworker stdout | 2023-01-17 04:53:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 45, 647286), True, datetime.datetime(2023, 1, 17, 4, 53, 45, 647286), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:53:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:53:49.537472+00:00 (in 2.599998 seconds) securityworker stdout | 2023-01-17 04:53:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:16 UTC)" (scheduled at 2023-01-17 04:53:46.936988+00:00) securityworker stdout | 2023-01-17 04:53:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:53:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:53:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:16.936988+00:00 (in 27.399004 seconds) securityworker stdout | 2023-01-17 04:53:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:19 UTC)" (scheduled at 2023-01-17 04:53:49.537472+00:00) securityworker stdout | 2023-01-17 04:53:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:53:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:53:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:53:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:53:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:53:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:53:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:53:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:53:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:53:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:53:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:53:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:53:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:53:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:53:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:53:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:21.348605+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 04:53:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:21 UTC)" (scheduled at 2023-01-17 04:53:51.348605+00:00) gcworker stdout | 2023-01-17 04:53:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721631349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:53:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:53:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:53:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:53:52,059 [233] [DEBUG] [app] Starting request: urn:request:dc5097ef-2834-4084-867c-a828b3d8ae4b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:52,059 [231] [DEBUG] [app] Starting request: urn:request:b3c88a91-ad0e-4473-aac4-285f63549ee2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:53:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:53:52,065 [230] [DEBUG] [app] Starting request: urn:request:2eac3a37-1d1d-4f98-8b01-a71ab09503c9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:52,065 [242] [DEBUG] [app] Starting request: urn:request:e00eb3f4-e448-4454-a78e-ea52c2c58cb2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:53:52,065 [230] [DEBUG] [app] Ending request: urn:request:2eac3a37-1d1d-4f98-8b01-a71ab09503c9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2eac3a37-1d1d-4f98-8b01-a71ab09503c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:53:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:53:52,065 [242] [DEBUG] [app] Ending request: urn:request:e00eb3f4-e448-4454-a78e-ea52c2c58cb2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e00eb3f4-e448-4454-a78e-ea52c2c58cb2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:53:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:53:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:53:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:53:52,071 [236] [DEBUG] [app] Starting request: urn:request:39bdceea-f8af-403a-bea7-806f1517f37a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:52,071 [235] [DEBUG] [app] Starting request: urn:request:9852532a-ce42-48f3-b259-87458f4de497 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:53:52,071 [236] [DEBUG] [app] Ending request: urn:request:39bdceea-f8af-403a-bea7-806f1517f37a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39bdceea-f8af-403a-bea7-806f1517f37a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:52,071 [235] [DEBUG] [app] Ending request: urn:request:9852532a-ce42-48f3-b259-87458f4de497 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9852532a-ce42-48f3-b259-87458f4de497', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:53:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:52,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:53:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:53:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:53:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:53:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:52,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:53:52,089 [233] [DEBUG] [app] Ending request: urn:request:dc5097ef-2834-4084-867c-a828b3d8ae4b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc5097ef-2834-4084-867c-a828b3d8ae4b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:52,089 [231] [DEBUG] [app] Ending request: urn:request:b3c88a91-ad0e-4473-aac4-285f63549ee2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b3c88a91-ad0e-4473-aac4-285f63549ee2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:53:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:53:52,089 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.032) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.032) gunicorn-web stdout | 2023-01-17 04:53:52,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:53:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:53:54,142 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:53:54,155 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:53:54,155 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:53:54,166 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:53:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:01.636362+00:00 (in 5.989509 seconds) notificationworker stdout | 2023-01-17 04:53:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:05 UTC)" (scheduled at 2023-01-17 04:53:55.646354+00:00) notificationworker stdout | 2023-01-17 04:53:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:53:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 53, 55, 647248), True, datetime.datetime(2023, 1, 17, 4, 53, 55, 647248), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:53:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:53:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:53:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:53:56,865 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:53:56,947 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:53:57,797 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:53:58,343 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:53:58,402 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:53:58,684 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:53:58,839 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:54:00,508 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:54:00,510 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:54:00,513 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:54:00,515 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:54:00,517 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:54:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:05.646354+00:00 (in 4.009491 seconds) notificationworker stdout | 2023-01-17 04:54:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:01 UTC)" (scheduled at 2023-01-17 04:54:01.636362+00:00) notificationworker stdout | 2023-01-17 04:54:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:54:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:54:01,856 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:54:02,240 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:54:03,387 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:54:03,390 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:54:03,671 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 04:54:04,176 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue chunkcleanupworker stdout | 2023-01-17 04:54:04,183 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} builder stdout | 2023-01-17 04:54:04,189 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:04,189 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:04,199 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 04:54:04,304 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:54:04,520 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:54:04,522 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:54:04,524 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:54:04,584 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:54:04,720 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:54:04,722 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:54:04,724 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:54:04,732 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:54:04,734 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:54:04,736 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:54:04,741 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} repositoryactioncounter stdout | 2023-01-17 04:54:04,743 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-registry stdout | 2023-01-17 04:54:04,745 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:54:04,747 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 04:54:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:54:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:19.346120+00:00 (in 14.408789 seconds) repositorygcworker stdout | 2023-01-17 04:54:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:04 UTC)" (scheduled at 2023-01-17 04:54:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:54:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:54:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 4, 937806), True, datetime.datetime(2023, 1, 17, 4, 54, 4, 937806), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:54:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:54:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:54:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 04:54:05,098 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:54:05,602 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 04:54:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:15.646354+00:00 (in 9.999499 seconds) notificationworker stdout | 2023-01-17 04:54:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:15 UTC)" (scheduled at 2023-01-17 04:54:05.646354+00:00) notificationworker stdout | 2023-01-17 04:54:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 5, 647274), True, datetime.datetime(2023, 1, 17, 4, 54, 5, 647274), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:15 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:54:05,772 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:54:05,881 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:54:07,060 [233] [DEBUG] [app] Starting request: urn:request:68371cd4-850d-4abf-8ef1-1bebed6dda4c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:07,060 [231] [DEBUG] [app] Starting request: urn:request:3bb44a91-c7a0-44de-baa4-ca5a4b866760 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:07,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:07,065 [243] [DEBUG] [app] Starting request: urn:request:e5ef9347-28b1-4be1-93d6-162b4ce62079 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,065 [239] [DEBUG] [app] Starting request: urn:request:03eccdce-1697-4c8c-b70f-ced19a168b1d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,066 [243] [DEBUG] [app] Ending request: urn:request:e5ef9347-28b1-4be1-93d6-162b4ce62079 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5ef9347-28b1-4be1-93d6-162b4ce62079', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,066 [239] [DEBUG] [app] Ending request: urn:request:03eccdce-1697-4c8c-b70f-ced19a168b1d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03eccdce-1697-4c8c-b70f-ced19a168b1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:54:07,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:54:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:07,070 [236] [DEBUG] [app] Starting request: urn:request:c92078fc-4ca8-4786-a5ef-9da9c0b708d5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:07,070 [235] [DEBUG] [app] Starting request: urn:request:61b88f89-c4b4-4568-b951-062821a35f95 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:07,070 [236] [DEBUG] [app] Ending request: urn:request:c92078fc-4ca8-4786-a5ef-9da9c0b708d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c92078fc-4ca8-4786-a5ef-9da9c0b708d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:07,071 [235] [DEBUG] [app] Ending request: urn:request:61b88f89-c4b4-4568-b951-062821a35f95 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61b88f89-c4b4-4568-b951-062821a35f95', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:54:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:07,088 [233] [DEBUG] [app] Ending request: urn:request:68371cd4-850d-4abf-8ef1-1bebed6dda4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:68371cd4-850d-4abf-8ef1-1bebed6dda4c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:07,088 [231] [DEBUG] [app] Ending request: urn:request:3bb44a91-c7a0-44de-baa4-ca5a4b866760 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3bb44a91-c7a0-44de-baa4-ca5a4b866760', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:54:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) buildlogsarchiver stdout | 2023-01-17 04:54:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:54:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:40.738568+00:00 (in 29.999541 seconds) buildlogsarchiver stdout | 2023-01-17 04:54:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:40 UTC)" (scheduled at 2023-01-17 04:54:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:54:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 54, 10, 739424), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:54:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:54:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:54:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:40 UTC)" executed successfully builder stdout | 2023-01-17 04:54:14,205 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:14,217 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:14,217 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:14,228 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:25.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 04:54:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:25 UTC)" (scheduled at 2023-01-17 04:54:15.646354+00:00) notificationworker stdout | 2023-01-17 04:54:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 15, 647189), True, datetime.datetime(2023, 1, 17, 4, 54, 15, 647189), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:19.537472+00:00 (in 2.600011 seconds) securityworker stdout | 2023-01-17 04:54:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:46 UTC)" (scheduled at 2023-01-17 04:54:16.936988+00:00) securityworker stdout | 2023-01-17 04:54:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:54:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:54:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:04.936835+00:00 (in 45.590233 seconds) repositorygcworker stdout | 2023-01-17 04:54:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:19 UTC)" (scheduled at 2023-01-17 04:54:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:54:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:54:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:46.936988+00:00 (in 27.399040 seconds) securityworker stdout | 2023-01-17 04:54:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:49 UTC)" (scheduled at 2023-01-17 04:54:19.537472+00:00) securityworker stdout | 2023-01-17 04:54:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:54:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:54:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:54:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:54:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:54:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:54:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:54:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:54:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:51.348605+00:00 (in 29.999509 seconds) gcworker stdout | 2023-01-17 04:54:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:51 UTC)" (scheduled at 2023-01-17 04:54:21.348605+00:00) gcworker stdout | 2023-01-17 04:54:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721661349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:54:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:54:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:54:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:54:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:54:22,059 [231] [DEBUG] [app] Starting request: urn:request:732e5c67-5678-4a72-ae63-59fa786c0c18 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:22,059 [236] [DEBUG] [app] Starting request: urn:request:46e7284c-5aeb-4849-9dfc-de4b30808b43 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:22,064 [243] [DEBUG] [app] Starting request: urn:request:8d0973c5-a792-4e55-be99-4735c5f135e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:22,064 [242] [DEBUG] [app] Starting request: urn:request:d8d1e398-cb7c-4b1c-b54f-887cb989c1f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:22,065 [243] [DEBUG] [app] Ending request: urn:request:8d0973c5-a792-4e55-be99-4735c5f135e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d0973c5-a792-4e55-be99-4735c5f135e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:22,065 [242] [DEBUG] [app] Ending request: urn:request:d8d1e398-cb7c-4b1c-b54f-887cb989c1f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8d1e398-cb7c-4b1c-b54f-887cb989c1f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:54:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:54:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:22,069 [233] [DEBUG] [app] Starting request: urn:request:a8a8ec1a-6669-438f-b1dc-0615af6e8ec8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:22,069 [231] [DEBUG] [app] Starting request: urn:request:16d7df94-5021-45a2-ad49-c5b297b1941a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:22,070 [233] [DEBUG] [app] Ending request: urn:request:a8a8ec1a-6669-438f-b1dc-0615af6e8ec8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a8a8ec1a-6669-438f-b1dc-0615af6e8ec8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:22,070 [231] [DEBUG] [app] Ending request: urn:request:16d7df94-5021-45a2-ad49-c5b297b1941a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16d7df94-5021-45a2-ad49-c5b297b1941a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:54:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:22,087 [231] [DEBUG] [app] Ending request: urn:request:732e5c67-5678-4a72-ae63-59fa786c0c18 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:732e5c67-5678-4a72-ae63-59fa786c0c18', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:22,087 [236] [DEBUG] [app] Ending request: urn:request:46e7284c-5aeb-4849-9dfc-de4b30808b43 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:46e7284c-5aeb-4849-9dfc-de4b30808b43', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 04:54:22,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:54:24,237 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:24,250 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:24,250 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:24,260 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:35.646354+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 04:54:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:35 UTC)" (scheduled at 2023-01-17 04:54:25.646354+00:00) notificationworker stdout | 2023-01-17 04:54:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 25, 647187), True, datetime.datetime(2023, 1, 17, 4, 54, 25, 647187), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:25,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:25,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:54:26,886 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:54:26,953 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:54:27,833 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:54:28,379 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:54:28,434 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:54:28,713 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:54:28,875 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:54:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:54:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:35.853396+00:00 (in 6.889308 seconds) exportactionlogsworker stdout | 2023-01-17 04:54:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:28 UTC)" (scheduled at 2023-01-17 04:54:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:54:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:54:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:28 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:54:30,515 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:54:30,518 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:54:30,520 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:54:30,523 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:54:30,525 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:54:31,871 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:54:32,272 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:54:33,408 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:54:33,422 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:54:33,692 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:54:34,218 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} builder stdout | 2023-01-17 04:54:34,270 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:34,283 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:34,283 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:34,294 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 04:54:34,318 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:54:34,526 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:54:34,529 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:54:34,531 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:54:34,619 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:54:34,728 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:54:34,731 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:54:34,734 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:54:34,739 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:54:34,743 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:54:34,745 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:54:34,749 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:54:34,753 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:54:34,755 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositoryactioncounter stdout | 2023-01-17 04:54:34,778 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 04:54:35,133 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 04:54:35,620 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 04:54:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:45.646354+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 04:54:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:45 UTC)" (scheduled at 2023-01-17 04:54:35.646354+00:00) notificationworker stdout | 2023-01-17 04:54:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 35, 647203), True, datetime.datetime(2023, 1, 17, 4, 54, 35, 647203), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:54:35,801 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} exportactionlogsworker stdout | 2023-01-17 04:54:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:54:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:28.963662+00:00 (in 53.109830 seconds) exportactionlogsworker stdout | 2023-01-17 04:54:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:35 UTC)" (scheduled at 2023-01-17 04:54:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:54:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:54:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 35, 854286), True, datetime.datetime(2023, 1, 17, 4, 54, 35, 854286), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:54:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:54:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:54:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:35 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 04:54:35,889 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:54:37,060 [235] [DEBUG] [app] Starting request: urn:request:e15693a4-fd9b-4beb-81a7-699e25cb7573 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:37,060 [236] [DEBUG] [app] Starting request: urn:request:6f255620-0d6d-45e3-8084-3358b17da0ae (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:37,065 [230] [DEBUG] [app] Starting request: urn:request:8ea16707-53fd-4738-8e7b-9ee64a872e23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:37,065 [243] [DEBUG] [app] Starting request: urn:request:e68d40fa-d150-46c9-9d98-dd4302a858d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:37,065 [230] [DEBUG] [app] Ending request: urn:request:8ea16707-53fd-4738-8e7b-9ee64a872e23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8ea16707-53fd-4738-8e7b-9ee64a872e23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:54:37,065 [243] [DEBUG] [app] Ending request: urn:request:e68d40fa-d150-46c9-9d98-dd4302a858d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e68d40fa-d150-46c9-9d98-dd4302a858d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:54:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:54:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:54:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:37,070 [231] [DEBUG] [app] Starting request: urn:request:2fe467bb-c7bc-426c-b5c2-8e09efe1c86c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:37,070 [233] [DEBUG] [app] Starting request: urn:request:706ea45a-d3d1-4b51-a9d7-6d52215800a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:37,070 [231] [DEBUG] [app] Ending request: urn:request:2fe467bb-c7bc-426c-b5c2-8e09efe1c86c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2fe467bb-c7bc-426c-b5c2-8e09efe1c86c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:37,070 [233] [DEBUG] [app] Ending request: urn:request:706ea45a-d3d1-4b51-a9d7-6d52215800a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:706ea45a-d3d1-4b51-a9d7-6d52215800a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:54:37,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:54:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:37,087 [236] [DEBUG] [app] Ending request: urn:request:6f255620-0d6d-45e3-8084-3358b17da0ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f255620-0d6d-45e3-8084-3358b17da0ae', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:37,087 [235] [DEBUG] [app] Ending request: urn:request:e15693a4-fd9b-4beb-81a7-699e25cb7573 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e15693a4-fd9b-4beb-81a7-699e25cb7573', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:37,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:54:37,087 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.027) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 04:54:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 04:54:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:54:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:37.537407+00:00 (in 0.280346 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:54:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:37 UTC)" (scheduled at 2023-01-17 04:54:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:54:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:54:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 37, 257509), True, datetime.datetime(2023, 1, 17, 4, 54, 37, 257509), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:54:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:54:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:54:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:54:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:54:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:37.256652+00:00 (in 59.718833 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:54:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:37 UTC)" (scheduled at 2023-01-17 04:54:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:54:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:54:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:54:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:54:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:10.738568+00:00 (in 29.999528 seconds) buildlogsarchiver stdout | 2023-01-17 04:54:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:10 UTC)" (scheduled at 2023-01-17 04:54:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:54:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 54, 40, 739465), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:54:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:54:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:54:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:54:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:54:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:42.941536+00:00 (in 1.792827 seconds) namespacegcworker stdout | 2023-01-17 04:54:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:41 UTC)" (scheduled at 2023-01-17 04:54:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:54:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:54:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 41, 149096), True, datetime.datetime(2023, 1, 17, 4, 54, 41, 149096), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:54:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:54:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:54:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:54:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:54:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:41.148245+00:00 (in 58.206243 seconds) namespacegcworker stdout | 2023-01-17 04:54:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:42 UTC)" (scheduled at 2023-01-17 04:54:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:54:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:54:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:55:42 UTC)" executed successfully builder stdout | 2023-01-17 04:54:44,304 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:44,316 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:44,316 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:44,327 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:55.646354+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 04:54:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:55 UTC)" (scheduled at 2023-01-17 04:54:45.646354+00:00) notificationworker stdout | 2023-01-17 04:54:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 45, 647207), True, datetime.datetime(2023, 1, 17, 4, 54, 45, 647207), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:54:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:54:49.537472+00:00 (in 2.599986 seconds) securityworker stdout | 2023-01-17 04:54:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:16 UTC)" (scheduled at 2023-01-17 04:54:46.936988+00:00) securityworker stdout | 2023-01-17 04:54:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:54:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:54:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:16.936988+00:00 (in 27.399010 seconds) securityworker stdout | 2023-01-17 04:54:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:19 UTC)" (scheduled at 2023-01-17 04:54:49.537472+00:00) securityworker stdout | 2023-01-17 04:54:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:54:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:54:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:54:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:54:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:54:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:54:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:54:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:54:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:54:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:54:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:54:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:54:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:54:49,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:54:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:54:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:21.348605+00:00 (in 29.999533 seconds) gcworker stdout | 2023-01-17 04:54:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:21 UTC)" (scheduled at 2023-01-17 04:54:51.348605+00:00) gcworker stdout | 2023-01-17 04:54:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721691349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:54:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:54:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:54:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:54:52,059 [231] [DEBUG] [app] Starting request: urn:request:bd2230c5-6df1-4b3c-bfe9-98a550cf0edc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:52,059 [236] [DEBUG] [app] Starting request: urn:request:983ca70f-e375-4b71-92b2-a81531615aea (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:54:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:54:52,064 [243] [DEBUG] [app] Starting request: urn:request:635fd6fd-567e-4367-8ab9-cbbe952b432b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:52,065 [241] [DEBUG] [app] Starting request: urn:request:c0fd3373-2006-46da-90f4-c9ee662e4824 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:54:52,065 [243] [DEBUG] [app] Ending request: urn:request:635fd6fd-567e-4367-8ab9-cbbe952b432b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:635fd6fd-567e-4367-8ab9-cbbe952b432b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:54:52,065 [241] [DEBUG] [app] Ending request: urn:request:c0fd3373-2006-46da-90f4-c9ee662e4824 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c0fd3373-2006-46da-90f4-c9ee662e4824', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:54:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:54:52,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:54:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:54:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:54:52,070 [235] [DEBUG] [app] Starting request: urn:request:0ff09ccd-0465-44c6-bb71-2b929d0a2a07 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:52,071 [231] [DEBUG] [app] Starting request: urn:request:3d1ed58b-cf22-4455-aef2-69b80f06f15f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:54:52,071 [231] [DEBUG] [app] Ending request: urn:request:3d1ed58b-cf22-4455-aef2-69b80f06f15f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d1ed58b-cf22-4455-aef2-69b80f06f15f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:52,071 [235] [DEBUG] [app] Ending request: urn:request:0ff09ccd-0465-44c6-bb71-2b929d0a2a07 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0ff09ccd-0465-44c6-bb71-2b929d0a2a07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:54:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:54:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:54:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:54:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:54:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:54:52,087 [231] [DEBUG] [app] Ending request: urn:request:bd2230c5-6df1-4b3c-bfe9-98a550cf0edc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd2230c5-6df1-4b3c-bfe9-98a550cf0edc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:52,087 [236] [DEBUG] [app] Ending request: urn:request:983ca70f-e375-4b71-92b2-a81531615aea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:983ca70f-e375-4b71-92b2-a81531615aea', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:54:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:54:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:54:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 04:54:54,334 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:54:54,346 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:54:54,347 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:54:54,357 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:54:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:01.636362+00:00 (in 5.989544 seconds) notificationworker stdout | 2023-01-17 04:54:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:05 UTC)" (scheduled at 2023-01-17 04:54:55.646354+00:00) notificationworker stdout | 2023-01-17 04:54:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:54:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 54, 55, 647244), True, datetime.datetime(2023, 1, 17, 4, 54, 55, 647244), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:54:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:54:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:54:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:54:56,922 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:54:56,982 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:54:57,869 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:54:58,390 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:54:58,471 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:54:58,749 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:54:58,910 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:55:00,523 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:55:00,525 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:55:00,528 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:55:00,530 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:55:00,532 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:55:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:05.646354+00:00 (in 4.009512 seconds) notificationworker stdout | 2023-01-17 04:55:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:01 UTC)" (scheduled at 2023-01-17 04:55:01.636362+00:00) notificationworker stdout | 2023-01-17 04:55:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:55:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:55:01,902 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:55:02,300 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:55:03,441 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:55:03,444 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:55:03,707 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:55:04,239 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:55:04,345 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:55:04,362 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:04,374 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:04,374 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:04,385 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 04:55:04,535 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:55:04,537 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:55:04,539 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:55:04,655 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:55:04,736 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:55:04,739 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:55:04,743 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:55:04,748 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:55:04,751 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:55:04,754 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:55:04,756 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:55:04,759 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:55:04,762 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositoryactioncounter stdout | 2023-01-17 04:55:04,812 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} repositorygcworker stdout | 2023-01-17 04:55:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:55:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:19.346120+00:00 (in 14.408843 seconds) repositorygcworker stdout | 2023-01-17 04:55:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:04 UTC)" (scheduled at 2023-01-17 04:55:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:55:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:55:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 4, 937735), True, datetime.datetime(2023, 1, 17, 4, 55, 4, 937735), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:55:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:55:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:55:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 04:55:05,168 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:55:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:15.646354+00:00 (in 9.999479 seconds) notificationworker stdout | 2023-01-17 04:55:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:15 UTC)" (scheduled at 2023-01-17 04:55:05.646354+00:00) notificationworker stdout | 2023-01-17 04:55:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 5, 647287), True, datetime.datetime(2023, 1, 17, 4, 55, 5, 647287), 0, 'notification/%', 50, 1, 0]) teamsyncworker stdout | 2023-01-17 04:55:05,655 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 04:55:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:15 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:55:05,836 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:55:05,923 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:55:07,060 [231] [DEBUG] [app] Starting request: urn:request:b16cc680-b078-47b3-8f00-a0ea3e0b9523 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:07,060 [235] [DEBUG] [app] Starting request: urn:request:cbb805a7-6512-4e12-a4fa-44341067362d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:07,065 [230] [DEBUG] [app] Starting request: urn:request:ea3c342b-bd69-41eb-bf7c-779bb27b054e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:07,065 [243] [DEBUG] [app] Starting request: urn:request:8172c84f-3cb4-4654-98a5-ad129768a3dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:07,065 [230] [DEBUG] [app] Ending request: urn:request:ea3c342b-bd69-41eb-bf7c-779bb27b054e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea3c342b-bd69-41eb-bf7c-779bb27b054e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:07,065 [243] [DEBUG] [app] Ending request: urn:request:8172c84f-3cb4-4654-98a5-ad129768a3dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8172c84f-3cb4-4654-98a5-ad129768a3dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 04:55:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:55:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:07,069 [231] [DEBUG] [app] Starting request: urn:request:c903caad-f46f-4268-adfd-33ada7f92024 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:07,070 [231] [DEBUG] [app] Ending request: urn:request:c903caad-f46f-4268-adfd-33ada7f92024 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c903caad-f46f-4268-adfd-33ada7f92024', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:55:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:07,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:07,072 [236] [DEBUG] [app] Starting request: urn:request:db4ab1e2-dcfe-4036-b810-5ea1ddbd045d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:07,072 [236] [DEBUG] [app] Ending request: urn:request:db4ab1e2-dcfe-4036-b810-5ea1ddbd045d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db4ab1e2-dcfe-4036-b810-5ea1ddbd045d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:07,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:07,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:07,087 [235] [DEBUG] [app] Ending request: urn:request:cbb805a7-6512-4e12-a4fa-44341067362d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cbb805a7-6512-4e12-a4fa-44341067362d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 04:55:07,088 [231] [DEBUG] [app] Ending request: urn:request:b16cc680-b078-47b3-8f00-a0ea3e0b9523 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b16cc680-b078-47b3-8f00-a0ea3e0b9523', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) buildlogsarchiver stdout | 2023-01-17 04:55:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:55:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:40.738568+00:00 (in 29.999512 seconds) buildlogsarchiver stdout | 2023-01-17 04:55:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:40 UTC)" (scheduled at 2023-01-17 04:55:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:55:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 55, 10, 739460), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:55:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:55:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:55:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:40 UTC)" executed successfully builder stdout | 2023-01-17 04:55:14,395 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:14,408 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:14,408 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:14,419 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:25.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 04:55:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:25 UTC)" (scheduled at 2023-01-17 04:55:15.646354+00:00) notificationworker stdout | 2023-01-17 04:55:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 15, 647289), True, datetime.datetime(2023, 1, 17, 4, 55, 15, 647289), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:19.537472+00:00 (in 2.599984 seconds) securityworker stdout | 2023-01-17 04:55:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:46 UTC)" (scheduled at 2023-01-17 04:55:16.936988+00:00) securityworker stdout | 2023-01-17 04:55:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:55:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:55:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:04.936835+00:00 (in 45.590241 seconds) repositorygcworker stdout | 2023-01-17 04:55:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:19 UTC)" (scheduled at 2023-01-17 04:55:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:55:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:55:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:46.936988+00:00 (in 27.399042 seconds) securityworker stdout | 2023-01-17 04:55:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:49 UTC)" (scheduled at 2023-01-17 04:55:19.537472+00:00) securityworker stdout | 2023-01-17 04:55:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:55:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:55:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:55:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:55:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:55:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:55:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:55:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:55:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:51.348605+00:00 (in 29.999566 seconds) gcworker stdout | 2023-01-17 04:55:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:51 UTC)" (scheduled at 2023-01-17 04:55:21.348605+00:00) gcworker stdout | 2023-01-17 04:55:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721721349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:55:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:55:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:55:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:55:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:55:22,059 [231] [DEBUG] [app] Starting request: urn:request:00499f4c-052d-4a6a-ae57-ed241740d385 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:22,059 [235] [DEBUG] [app] Starting request: urn:request:c49345a6-695f-4baf-967d-392f4832e13a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:22,064 [230] [DEBUG] [app] Starting request: urn:request:c6d3b8f2-0101-4e5e-8ad8-01f001c7e3f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:22,065 [243] [DEBUG] [app] Starting request: urn:request:f0e66ac4-fb24-485e-853a-8562900ba22e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:22,065 [230] [DEBUG] [app] Ending request: urn:request:c6d3b8f2-0101-4e5e-8ad8-01f001c7e3f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c6d3b8f2-0101-4e5e-8ad8-01f001c7e3f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:55:22,065 [243] [DEBUG] [app] Ending request: urn:request:f0e66ac4-fb24-485e-853a-8562900ba22e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0e66ac4-fb24-485e-853a-8562900ba22e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:55:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 04:55:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:22,070 [233] [DEBUG] [app] Starting request: urn:request:9a5824d3-fabe-4271-9847-1e79931a9143 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:22,070 [231] [DEBUG] [app] Starting request: urn:request:c005a23d-0dfb-46ef-9184-db4db5eb54da (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:22,070 [233] [DEBUG] [app] Ending request: urn:request:9a5824d3-fabe-4271-9847-1e79931a9143 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9a5824d3-fabe-4271-9847-1e79931a9143', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:22,070 [231] [DEBUG] [app] Ending request: urn:request:c005a23d-0dfb-46ef-9184-db4db5eb54da (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c005a23d-0dfb-46ef-9184-db4db5eb54da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:22,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:22,087 [231] [DEBUG] [app] Ending request: urn:request:00499f4c-052d-4a6a-ae57-ed241740d385 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:00499f4c-052d-4a6a-ae57-ed241740d385', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:22,087 [235] [DEBUG] [app] Ending request: urn:request:c49345a6-695f-4baf-967d-392f4832e13a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c49345a6-695f-4baf-967d-392f4832e13a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:22,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:55:22,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 04:55:24,429 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:24,442 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:24,442 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:24,452 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:35.646354+00:00 (in 9.999450 seconds) notificationworker stdout | 2023-01-17 04:55:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:35 UTC)" (scheduled at 2023-01-17 04:55:25.646354+00:00) notificationworker stdout | 2023-01-17 04:55:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 25, 647409), True, datetime.datetime(2023, 1, 17, 4, 55, 25, 647409), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:55:26,958 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:55:27,017 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:55:27,903 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:55:28,426 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:55:28,507 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:55:28,756 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:55:28,926 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} exportactionlogsworker stdout | 2023-01-17 04:55:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:55:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:35.853396+00:00 (in 6.889297 seconds) exportactionlogsworker stdout | 2023-01-17 04:55:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:28 UTC)" (scheduled at 2023-01-17 04:55:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:55:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:55:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:28 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:55:30,529 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:55:30,532 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:55:30,534 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:55:30,537 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:55:30,539 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:55:31,912 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:55:32,330 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:55:33,463 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:55:33,479 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:55:33,719 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:55:34,257 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:55:34,378 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 04:55:34,462 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:34,475 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:34,475 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:34,485 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 04:55:34,542 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:55:34,544 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:55:34,546 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:55:34,685 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:55:34,743 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:55:34,747 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:55:34,753 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:55:34,755 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:55:34,758 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:55:34,761 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:55:34,763 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:55:34,765 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:55:34,769 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositoryactioncounter stdout | 2023-01-17 04:55:34,847 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 04:55:35,196 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:55:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:45.646354+00:00 (in 9.999450 seconds) notificationworker stdout | 2023-01-17 04:55:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:45 UTC)" (scheduled at 2023-01-17 04:55:35.646354+00:00) notificationworker stdout | 2023-01-17 04:55:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 35, 647368), True, datetime.datetime(2023, 1, 17, 4, 55, 35, 647368), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:45 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:55:35,681 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:55:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:55:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:28.963662+00:00 (in 53.109810 seconds) exportactionlogsworker stdout | 2023-01-17 04:55:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:35 UTC)" (scheduled at 2023-01-17 04:55:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:55:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:55:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 35, 854245), True, datetime.datetime(2023, 1, 17, 4, 55, 35, 854245), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:55:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:55:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:55:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:55:35,870 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:55:35,958 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:55:37,060 [233] [DEBUG] [app] Starting request: urn:request:b5371c07-2ca2-4cf6-ac3f-eb3815c41592 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:37,061 [236] [DEBUG] [app] Starting request: urn:request:38bf4f92-320b-40fb-8c74-d7af18551e24 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:37,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:37,066 [230] [DEBUG] [app] Starting request: urn:request:b2a3c269-471c-4872-89fa-0d563de242ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:37,066 [243] [DEBUG] [app] Starting request: urn:request:d89ff42d-a59b-4bc7-a82e-fd641a5ab7bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:37,066 [230] [DEBUG] [app] Ending request: urn:request:b2a3c269-471c-4872-89fa-0d563de242ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2a3c269-471c-4872-89fa-0d563de242ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:55:37,066 [243] [DEBUG] [app] Ending request: urn:request:d89ff42d-a59b-4bc7-a82e-fd641a5ab7bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d89ff42d-a59b-4bc7-a82e-fd641a5ab7bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:37,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:37,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:55:37,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:37,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:37,071 [231] [DEBUG] [app] Starting request: urn:request:f94d4b19-0dcb-4ff4-be22-b4aed9e63c0f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:37,072 [233] [DEBUG] [app] Starting request: urn:request:ca872560-cd39-4e7e-a086-6a79a07836b4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:37,072 [231] [DEBUG] [app] Ending request: urn:request:f94d4b19-0dcb-4ff4-be22-b4aed9e63c0f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f94d4b19-0dcb-4ff4-be22-b4aed9e63c0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:37,072 [233] [DEBUG] [app] Ending request: urn:request:ca872560-cd39-4e7e-a086-6a79a07836b4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ca872560-cd39-4e7e-a086-6a79a07836b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:37,073 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:37,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:37,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:37,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:37,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:37,089 [233] [DEBUG] [app] Ending request: urn:request:b5371c07-2ca2-4cf6-ac3f-eb3815c41592 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5371c07-2ca2-4cf6-ac3f-eb3815c41592', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:37,089 [236] [DEBUG] [app] Ending request: urn:request:38bf4f92-320b-40fb-8c74-d7af18551e24 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38bf4f92-320b-40fb-8c74-d7af18551e24', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:37,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:37,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:37,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:55:37,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 04:55:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:55:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:37.537407+00:00 (in 0.280327 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:55:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:37 UTC)" (scheduled at 2023-01-17 04:55:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:55:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:55:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 37, 257493), True, datetime.datetime(2023, 1, 17, 4, 55, 37, 257493), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:55:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:55:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:55:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:55:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:55:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:23.537048+00:00 (in 45.999242 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:55:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:37 UTC)" (scheduled at 2023-01-17 04:55:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:55:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:55:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:55:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:55:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:10.738568+00:00 (in 29.999593 seconds) buildlogsarchiver stdout | 2023-01-17 04:55:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:10 UTC)" (scheduled at 2023-01-17 04:55:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:55:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 55, 40, 739382), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:55:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:55:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:55:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:55:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:55:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:42.941536+00:00 (in 1.792863 seconds) namespacegcworker stdout | 2023-01-17 04:55:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:41 UTC)" (scheduled at 2023-01-17 04:55:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:55:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:55:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 41, 149095), True, datetime.datetime(2023, 1, 17, 4, 55, 41, 149095), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:55:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:55:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:55:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:55:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:55:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:41.148245+00:00 (in 58.206212 seconds) namespacegcworker stdout | 2023-01-17 04:55:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:42 UTC)" (scheduled at 2023-01-17 04:55:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:55:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:55:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:56:42 UTC)" executed successfully builder stdout | 2023-01-17 04:55:44,496 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:44,508 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:44,509 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:44,519 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:55.646354+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 04:55:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:55 UTC)" (scheduled at 2023-01-17 04:55:45.646354+00:00) notificationworker stdout | 2023-01-17 04:55:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 45, 647208), True, datetime.datetime(2023, 1, 17, 4, 55, 45, 647208), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:55:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:55:49.537472+00:00 (in 2.600000 seconds) securityworker stdout | 2023-01-17 04:55:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:16 UTC)" (scheduled at 2023-01-17 04:55:46.936988+00:00) securityworker stdout | 2023-01-17 04:55:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:55:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:55:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:16.936988+00:00 (in 27.399041 seconds) securityworker stdout | 2023-01-17 04:55:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:19 UTC)" (scheduled at 2023-01-17 04:55:49.537472+00:00) securityworker stdout | 2023-01-17 04:55:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:55:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:55:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:55:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:55:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:55:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:55:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:55:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:55:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:55:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:55:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:55:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:55:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:55:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:55:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:55:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:21.348605+00:00 (in 29.999578 seconds) gcworker stdout | 2023-01-17 04:55:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:21 UTC)" (scheduled at 2023-01-17 04:55:51.348605+00:00) gcworker stdout | 2023-01-17 04:55:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721751349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:55:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:55:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:55:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:55:52,060 [231] [DEBUG] [app] Starting request: urn:request:6b338e66-a0ab-44fc-ba6f-b12280e30b21 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:52,060 [235] [DEBUG] [app] Starting request: urn:request:b1bde449-648e-48de-97a5-c7d4aeb759f4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:55:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:55:52,064 [230] [DEBUG] [app] Starting request: urn:request:68a92734-9259-4f4f-915e-c4d2408c4271 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:52,065 [241] [DEBUG] [app] Starting request: urn:request:ab2fa7a6-ac29-48c6-a9dc-89517d1406a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:55:52,065 [230] [DEBUG] [app] Ending request: urn:request:68a92734-9259-4f4f-915e-c4d2408c4271 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68a92734-9259-4f4f-915e-c4d2408c4271', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:52,065 [241] [DEBUG] [app] Ending request: urn:request:ab2fa7a6-ac29-48c6-a9dc-89517d1406a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab2fa7a6-ac29-48c6-a9dc-89517d1406a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:55:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:55:52,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:55:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:55:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:52,070 [235] [DEBUG] [app] Starting request: urn:request:fd119e33-e33d-49aa-b0ca-ff24219e8f1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:52,070 [235] [DEBUG] [app] Ending request: urn:request:fd119e33-e33d-49aa-b0ca-ff24219e8f1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd119e33-e33d-49aa-b0ca-ff24219e8f1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:55:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:55:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:55:52,072 [236] [DEBUG] [app] Starting request: urn:request:1896d972-6169-4a2d-be02-1a6744b752cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:55:52,073 [236] [DEBUG] [app] Ending request: urn:request:1896d972-6169-4a2d-be02-1a6744b752cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1896d972-6169-4a2d-be02-1a6744b752cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:55:52,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:55:52,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:55:52,074 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:55:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:52,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:55:52,087 [231] [DEBUG] [app] Ending request: urn:request:6b338e66-a0ab-44fc-ba6f-b12280e30b21 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b338e66-a0ab-44fc-ba6f-b12280e30b21', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 04:55:52,089 [235] [DEBUG] [app] Ending request: urn:request:b1bde449-648e-48de-97a5-c7d4aeb759f4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b1bde449-648e-48de-97a5-c7d4aeb759f4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:55:52,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:55:52,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) builder stdout | 2023-01-17 04:55:54,525 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:55:54,537 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:55:54,538 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:55:54,548 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:55:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:01.636362+00:00 (in 5.989564 seconds) notificationworker stdout | 2023-01-17 04:55:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:05 UTC)" (scheduled at 2023-01-17 04:55:55.646354+00:00) notificationworker stdout | 2023-01-17 04:55:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:55:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 55, 55, 647217), True, datetime.datetime(2023, 1, 17, 4, 55, 55, 647217), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:55:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:55:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:55:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:55:56,994 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:55:57,035 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:55:57,939 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:55:58,445 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:55:58,540 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:55:58,792 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:55:58,950 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:56:00,537 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:56:00,540 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:56:00,542 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:56:00,544 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:56:00,546 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:56:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:05.646354+00:00 (in 4.009527 seconds) notificationworker stdout | 2023-01-17 04:56:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:01 UTC)" (scheduled at 2023-01-17 04:56:01.636362+00:00) notificationworker stdout | 2023-01-17 04:56:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:56:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:56:01,927 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:56:02,365 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:56:03,486 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:56:03,498 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:56:03,726 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:56:04,290 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:56:04,388 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:56:04,549 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:56:04,552 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:56:04,554 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} builder stdout | 2023-01-17 04:56:04,558 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:04,571 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:04,571 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:04,581 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:56:04,692 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:56:04,750 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:56:04,753 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:56:04,758 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:56:04,761 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:56:04,765 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:56:04,767 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:56:04,770 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:56:04,772 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:56:04,776 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositoryactioncounter stdout | 2023-01-17 04:56:04,870 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} repositorygcworker stdout | 2023-01-17 04:56:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:56:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:17.345601+00:00 (in 12.408345 seconds) repositorygcworker stdout | 2023-01-17 04:56:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:04 UTC)" (scheduled at 2023-01-17 04:56:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:56:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:56:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 4, 937719), True, datetime.datetime(2023, 1, 17, 4, 56, 4, 937719), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:56:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:56:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:56:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 04:56:05,222 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:56:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:15.646354+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 04:56:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:15 UTC)" (scheduled at 2023-01-17 04:56:05.646354+00:00) notificationworker stdout | 2023-01-17 04:56:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 5, 647184), True, datetime.datetime(2023, 1, 17, 4, 56, 5, 647184), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:15 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:56:05,717 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:56:05,905 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:56:05,987 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:56:07,060 [231] [DEBUG] [app] Starting request: urn:request:7b61ca84-45b2-405e-98e3-fb522738da1a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:07,060 [235] [DEBUG] [app] Starting request: urn:request:a5f064e6-c45b-4110-9c6c-d2e383fc1935 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:07,064 [230] [DEBUG] [app] Starting request: urn:request:243b84b2-f92c-4b05-8d73-fbac10bae53b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:07,065 [230] [DEBUG] [app] Ending request: urn:request:243b84b2-f92c-4b05-8d73-fbac10bae53b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:243b84b2-f92c-4b05-8d73-fbac10bae53b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:56:07,065 [239] [DEBUG] [app] Starting request: urn:request:d1fe2b54-7fa4-46f7-a960-0361efa11511 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:56:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:56:07,066 [239] [DEBUG] [app] Ending request: urn:request:d1fe2b54-7fa4-46f7-a960-0361efa11511 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d1fe2b54-7fa4-46f7-a960-0361efa11511', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:56:07,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:56:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:07,070 [236] [DEBUG] [app] Starting request: urn:request:86978f41-db9c-4a32-acea-29fcdda91bd1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:07,070 [231] [DEBUG] [app] Starting request: urn:request:af37271a-025c-4aa8-8e1e-5007b2745143 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:07,070 [236] [DEBUG] [app] Ending request: urn:request:86978f41-db9c-4a32-acea-29fcdda91bd1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:86978f41-db9c-4a32-acea-29fcdda91bd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:07,070 [231] [DEBUG] [app] Ending request: urn:request:af37271a-025c-4aa8-8e1e-5007b2745143 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af37271a-025c-4aa8-8e1e-5007b2745143', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:07,087 [231] [DEBUG] [app] Ending request: urn:request:7b61ca84-45b2-405e-98e3-fb522738da1a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7b61ca84-45b2-405e-98e3-fb522738da1a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:07,087 [235] [DEBUG] [app] Ending request: urn:request:a5f064e6-c45b-4110-9c6c-d2e383fc1935 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5f064e6-c45b-4110-9c6c-d2e383fc1935', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:56:07,087 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 04:56:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 04:56:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:56:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:40.738568+00:00 (in 29.999571 seconds) buildlogsarchiver stdout | 2023-01-17 04:56:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:40 UTC)" (scheduled at 2023-01-17 04:56:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:56:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 56, 10, 739379), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:56:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:56:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:56:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:40 UTC)" executed successfully builder stdout | 2023-01-17 04:56:14,592 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:14,611 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:14,611 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:14,621 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:25.646354+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 04:56:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:25 UTC)" (scheduled at 2023-01-17 04:56:15.646354+00:00) notificationworker stdout | 2023-01-17 04:56:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 15, 647222), True, datetime.datetime(2023, 1, 17, 4, 56, 15, 647222), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:19.537472+00:00 (in 2.599996 seconds) securityworker stdout | 2023-01-17 04:56:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:46 UTC)" (scheduled at 2023-01-17 04:56:16.936988+00:00) securityworker stdout | 2023-01-17 04:56:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:56:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:56:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:19.346120+00:00 (in 2.000054 seconds) repositorygcworker stdout | 2023-01-17 04:56:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:17 UTC)" (scheduled at 2023-01-17 04:56:17.345601+00:00) repositorygcworker stdout | 2023-01-17 04:56:17,346 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 56, 17, 346409), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:56:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 17, 346409), True, datetime.datetime(2023, 1, 17, 4, 56, 17, 346409), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:56:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 17, 346409), True, datetime.datetime(2023, 1, 17, 4, 56, 17, 346409), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 4, 56, 17, 346409), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 04:56:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:56:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:56:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:56:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:04.936835+00:00 (in 45.590292 seconds) repositorygcworker stdout | 2023-01-17 04:56:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:19 UTC)" (scheduled at 2023-01-17 04:56:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:56:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:56:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:46.936988+00:00 (in 27.399088 seconds) securityworker stdout | 2023-01-17 04:56:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:49 UTC)" (scheduled at 2023-01-17 04:56:19.537472+00:00) securityworker stdout | 2023-01-17 04:56:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:56:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:56:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:56:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:56:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:56:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:56:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:56:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:56:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:51.348605+00:00 (in 29.999563 seconds) gcworker stdout | 2023-01-17 04:56:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:51 UTC)" (scheduled at 2023-01-17 04:56:21.348605+00:00) gcworker stdout | 2023-01-17 04:56:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721781349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:56:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:56:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:56:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:56:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:56:22,060 [231] [DEBUG] [app] Starting request: urn:request:d1c0b028-ea59-4a05-b045-a71e246aeef5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:22,060 [236] [DEBUG] [app] Starting request: urn:request:38462142-b794-46d7-ade9-4e59ff2b0cf0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:22,065 [243] [DEBUG] [app] Starting request: urn:request:a39cae17-d8b5-415c-9c0b-c3ea47ba0f3a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:22,065 [230] [DEBUG] [app] Starting request: urn:request:4902e6e4-d401-4fac-9708-475e0d0c5840 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:22,066 [243] [DEBUG] [app] Ending request: urn:request:a39cae17-d8b5-415c-9c0b-c3ea47ba0f3a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a39cae17-d8b5-415c-9c0b-c3ea47ba0f3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:56:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:56:22,066 [230] [DEBUG] [app] Ending request: urn:request:4902e6e4-d401-4fac-9708-475e0d0c5840 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4902e6e4-d401-4fac-9708-475e0d0c5840', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:56:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:56:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:22,071 [235] [DEBUG] [app] Starting request: urn:request:b8900847-5e93-48ac-9a4a-b993b25e764f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:22,071 [233] [DEBUG] [app] Starting request: urn:request:e1b22140-0177-435d-a625-8c536308660c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:22,071 [235] [DEBUG] [app] Ending request: urn:request:b8900847-5e93-48ac-9a4a-b993b25e764f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8900847-5e93-48ac-9a4a-b993b25e764f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:22,072 [233] [DEBUG] [app] Ending request: urn:request:e1b22140-0177-435d-a625-8c536308660c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e1b22140-0177-435d-a625-8c536308660c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:22,088 [231] [DEBUG] [app] Ending request: urn:request:d1c0b028-ea59-4a05-b045-a71e246aeef5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1c0b028-ea59-4a05-b045-a71e246aeef5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:22,088 [236] [DEBUG] [app] Ending request: urn:request:38462142-b794-46d7-ade9-4e59ff2b0cf0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38462142-b794-46d7-ade9-4e59ff2b0cf0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:56:22,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 04:56:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 04:56:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:56:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:37.256652+00:00 (in 13.719166 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:56:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:23 UTC)" (scheduled at 2023-01-17 04:56:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:56:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 56, 23, 537869), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:56:23,549 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 23, 537869), True, datetime.datetime(2023, 1, 17, 4, 56, 23, 537869), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:56:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 23, 537869), True, datetime.datetime(2023, 1, 17, 4, 56, 23, 537869), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 4, 56, 23, 537869), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 04:56:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:56:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:23 UTC)" executed successfully builder stdout | 2023-01-17 04:56:24,622 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:24,634 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:24,634 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:24,645 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:35.646354+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 04:56:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:35 UTC)" (scheduled at 2023-01-17 04:56:25.646354+00:00) notificationworker stdout | 2023-01-17 04:56:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 25, 647162), True, datetime.datetime(2023, 1, 17, 4, 56, 25, 647162), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:56:27,023 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:56:27,070 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:56:27,946 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:56:28,451 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:56:28,549 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:56:28,823 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} exportactionlogsworker stdout | 2023-01-17 04:56:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:56:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:35.853396+00:00 (in 6.889298 seconds) exportactionlogsworker stdout | 2023-01-17 04:56:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:28 UTC)" (scheduled at 2023-01-17 04:56:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:56:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:56:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:28 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:28,965 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:56:30,544 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:56:30,547 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:56:30,549 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:56:30,552 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:56:30,554 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:56:31,951 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:56:32,381 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:56:33,520 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:56:33,532 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:56:33,757 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:56:34,297 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:56:34,422 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:56:34,556 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:56:34,558 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:56:34,561 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} builder stdout | 2023-01-17 04:56:34,655 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:34,667 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:34,667 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:34,678 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:56:34,703 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:56:34,757 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:56:34,759 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:56:34,766 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:56:34,768 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:56:34,771 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:56:34,774 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:56:34,776 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:56:34,783 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:56:34,786 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositoryactioncounter stdout | 2023-01-17 04:56:34,889 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 04:56:35,239 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:56:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:45.646354+00:00 (in 9.999574 seconds) notificationworker stdout | 2023-01-17 04:56:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:45 UTC)" (scheduled at 2023-01-17 04:56:35.646354+00:00) notificationworker stdout | 2023-01-17 04:56:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 35, 647219), True, datetime.datetime(2023, 1, 17, 4, 56, 35, 647219), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:45 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:56:35,738 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:56:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:56:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:54.963413+00:00 (in 19.109595 seconds) exportactionlogsworker stdout | 2023-01-17 04:56:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:35 UTC)" (scheduled at 2023-01-17 04:56:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:56:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:56:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 35, 854185), True, datetime.datetime(2023, 1, 17, 4, 56, 35, 854185), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:56:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:56:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:56:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:56:35,940 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:56:35,994 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:56:37,060 [235] [DEBUG] [app] Starting request: urn:request:09dae090-617a-4f3c-a6d9-9654e65f1ce4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:37,060 [231] [DEBUG] [app] Starting request: urn:request:26d08194-e0d6-4790-82ef-731e1d99010c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:37,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:37,066 [230] [DEBUG] [app] Starting request: urn:request:97af94e0-8f1d-4db2-931a-351e3567fdef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:37,066 [230] [DEBUG] [app] Ending request: urn:request:97af94e0-8f1d-4db2-931a-351e3567fdef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97af94e0-8f1d-4db2-931a-351e3567fdef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:56:37,066 [243] [DEBUG] [app] Starting request: urn:request:d35c14eb-f933-4f60-8ac8-2a37e0cc570d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:56:37,066 [243] [DEBUG] [app] Ending request: urn:request:d35c14eb-f933-4f60-8ac8-2a37e0cc570d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d35c14eb-f933-4f60-8ac8-2a37e0cc570d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:56:37,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:56:37,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:37,070 [236] [DEBUG] [app] Starting request: urn:request:1c974f8f-9a23-4955-9fdf-90cd133bee72 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:37,071 [236] [DEBUG] [app] Ending request: urn:request:1c974f8f-9a23-4955-9fdf-90cd133bee72 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1c974f8f-9a23-4955-9fdf-90cd133bee72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:37,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:56:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:37,072 [233] [DEBUG] [app] Starting request: urn:request:af5c0608-db08-4e9e-83f5-9f21aa96fdab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:37,073 [233] [DEBUG] [app] Ending request: urn:request:af5c0608-db08-4e9e-83f5-9f21aa96fdab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af5c0608-db08-4e9e-83f5-9f21aa96fdab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:37,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:37,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:37,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:37,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:37,088 [231] [DEBUG] [app] Ending request: urn:request:26d08194-e0d6-4790-82ef-731e1d99010c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26d08194-e0d6-4790-82ef-731e1d99010c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 04:56:37,089 [235] [DEBUG] [app] Ending request: urn:request:09dae090-617a-4f3c-a6d9-9654e65f1ce4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09dae090-617a-4f3c-a6d9-9654e65f1ce4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:37,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:37,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 04:56:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:56:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:37.537407+00:00 (in 0.280258 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:56:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:37 UTC)" (scheduled at 2023-01-17 04:56:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:56:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:56:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 37, 257613), True, datetime.datetime(2023, 1, 17, 4, 56, 37, 257613), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:56:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:56:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:56:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:56:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:56:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:37.256652+00:00 (in 59.718789 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:56:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:37 UTC)" (scheduled at 2023-01-17 04:56:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:56:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:56:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:56:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:56:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:10.738568+00:00 (in 29.999526 seconds) buildlogsarchiver stdout | 2023-01-17 04:56:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:10 UTC)" (scheduled at 2023-01-17 04:56:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:56:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 56, 40, 739455), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:56:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:56:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:56:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:56:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:56:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:42.941536+00:00 (in 1.792833 seconds) namespacegcworker stdout | 2023-01-17 04:56:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:41 UTC)" (scheduled at 2023-01-17 04:56:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:56:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:56:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 41, 149158), True, datetime.datetime(2023, 1, 17, 4, 56, 41, 149158), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:56:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:56:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:56:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:56:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:56:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:06.941185+00:00 (in 23.999173 seconds) namespacegcworker stdout | 2023-01-17 04:56:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:42 UTC)" (scheduled at 2023-01-17 04:56:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:56:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:56:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:57:42 UTC)" executed successfully builder stdout | 2023-01-17 04:56:44,681 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:44,692 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:44,693 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:44,703 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:55.646354+00:00 (in 9.999552 seconds) notificationworker stdout | 2023-01-17 04:56:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:55 UTC)" (scheduled at 2023-01-17 04:56:45.646354+00:00) notificationworker stdout | 2023-01-17 04:56:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 45, 647244), True, datetime.datetime(2023, 1, 17, 4, 56, 45, 647244), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:56:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:56:49.537472+00:00 (in 2.600037 seconds) securityworker stdout | 2023-01-17 04:56:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:16 UTC)" (scheduled at 2023-01-17 04:56:46.936988+00:00) securityworker stdout | 2023-01-17 04:56:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:56:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:56:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:16.936988+00:00 (in 27.399052 seconds) securityworker stdout | 2023-01-17 04:56:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:19 UTC)" (scheduled at 2023-01-17 04:56:49.537472+00:00) securityworker stdout | 2023-01-17 04:56:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:56:49,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:49,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:56:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:56:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:56:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:56:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:56:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:56:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:56:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:56:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:56:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:56:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:56:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:56:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:56:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:56:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:21.348605+00:00 (in 29.999597 seconds) gcworker stdout | 2023-01-17 04:56:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:21 UTC)" (scheduled at 2023-01-17 04:56:51.348605+00:00) gcworker stdout | 2023-01-17 04:56:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721811349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:56:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:56:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:56:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:56:52,060 [236] [DEBUG] [app] Starting request: urn:request:8f3000d3-4ea4-47aa-b7c5-cbb5c6fc954d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:52,060 [231] [DEBUG] [app] Starting request: urn:request:10cb95ed-e62c-4acb-ac91-e3480a77462d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:56:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:56:52,065 [243] [DEBUG] [app] Starting request: urn:request:8477208e-f51f-450a-99c5-9c91b8d70258 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:52,065 [230] [DEBUG] [app] Starting request: urn:request:421f03a5-83bd-4073-b095-a141cbe0c9fc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:56:52,065 [243] [DEBUG] [app] Ending request: urn:request:8477208e-f51f-450a-99c5-9c91b8d70258 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8477208e-f51f-450a-99c5-9c91b8d70258', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:56:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:56:52,066 [230] [DEBUG] [app] Ending request: urn:request:421f03a5-83bd-4073-b095-a141cbe0c9fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:421f03a5-83bd-4073-b095-a141cbe0c9fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:56:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:56:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:56:52,070 [235] [DEBUG] [app] Starting request: urn:request:dd938a46-a575-47ee-97dc-96502b7219c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:52,070 [231] [DEBUG] [app] Starting request: urn:request:8348742e-1c32-4794-8e3f-ff9130bf56e6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:56:52,070 [235] [DEBUG] [app] Ending request: urn:request:dd938a46-a575-47ee-97dc-96502b7219c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd938a46-a575-47ee-97dc-96502b7219c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:56:52,070 [231] [DEBUG] [app] Ending request: urn:request:8348742e-1c32-4794-8e3f-ff9130bf56e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8348742e-1c32-4794-8e3f-ff9130bf56e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:56:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:56:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:56:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:56:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:56:52,087 [236] [DEBUG] [app] Ending request: urn:request:8f3000d3-4ea4-47aa-b7c5-cbb5c6fc954d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f3000d3-4ea4-47aa-b7c5-cbb5c6fc954d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:52,087 [231] [DEBUG] [app] Ending request: urn:request:10cb95ed-e62c-4acb-ac91-e3480a77462d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10cb95ed-e62c-4acb-ac91-e3480a77462d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:56:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:56:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:56:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) builder stdout | 2023-01-17 04:56:54,706 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:56:54,719 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:56:54,719 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:56:54,729 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:56:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:56:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:28.963662+00:00 (in 33.999782 seconds) exportactionlogsworker stdout | 2023-01-17 04:56:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:54 UTC)" (scheduled at 2023-01-17 04:56:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 04:56:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 56, 54, 964276), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:56:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 54, 964276), True, datetime.datetime(2023, 1, 17, 4, 56, 54, 964276), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:56:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 56, 54, 964276), True, datetime.datetime(2023, 1, 17, 4, 56, 54, 964276), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 4, 56, 54, 964276), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 04:56:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:56:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:01:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:56:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:56:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:01.636362+00:00 (in 5.989587 seconds) notificationworker stdout | 2023-01-17 04:56:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:05 UTC)" (scheduled at 2023-01-17 04:56:55.646354+00:00) notificationworker stdout | 2023-01-17 04:56:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:56:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 56, 55, 647195), True, datetime.datetime(2023, 1, 17, 4, 56, 55, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:56:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:56:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:56:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:56:57,033 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:56:57,089 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:56:57,966 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:56:58,486 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:56:58,586 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:56:58,839 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:56:59,000 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:57:00,553 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:57:00,555 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:57:00,558 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:57:00,560 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:57:00,562 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:57:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:05.646354+00:00 (in 4.009528 seconds) notificationworker stdout | 2023-01-17 04:57:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:01 UTC)" (scheduled at 2023-01-17 04:57:01.636362+00:00) notificationworker stdout | 2023-01-17 04:57:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:57:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:57:01,987 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:57:02,390 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:57:03,538 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:57:03,567 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:57:03,793 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:57:04,333 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:57:04,459 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:57:04,563 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:57:04,566 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:57:04,568 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:57:04,726 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} builder stdout | 2023-01-17 04:57:04,739 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:04,751 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:04,751 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:04,762 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 04:57:04,768 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:57:04,771 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:57:04,773 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:57:04,775 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:57:04,781 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:57:04,783 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:57:04,786 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:57:04,791 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:57:04,795 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositoryactioncounter stdout | 2023-01-17 04:57:04,923 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} repositorygcworker stdout | 2023-01-17 04:57:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:57:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:19.346120+00:00 (in 14.408870 seconds) repositorygcworker stdout | 2023-01-17 04:57:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:04 UTC)" (scheduled at 2023-01-17 04:57:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:57:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:57:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 4, 937637), True, datetime.datetime(2023, 1, 17, 4, 57, 4, 937637), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:57:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:57:04,948 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:57:04,948 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:04 UTC)" executed successfully globalpromstats stdout | 2023-01-17 04:57:05,275 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:57:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:15.646354+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 04:57:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:15 UTC)" (scheduled at 2023-01-17 04:57:05.646354+00:00) notificationworker stdout | 2023-01-17 04:57:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 5, 647338), True, datetime.datetime(2023, 1, 17, 4, 57, 5, 647338), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:15 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:57:05,745 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:57:05,975 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:57:06,009 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} namespacegcworker stdout | 2023-01-17 04:57:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:57:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:41.148245+00:00 (in 34.206559 seconds) namespacegcworker stdout | 2023-01-17 04:57:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:02:06 UTC)" (scheduled at 2023-01-17 04:57:06.941185+00:00) namespacegcworker stdout | 2023-01-17 04:57:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 57, 6, 942072), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:57:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 57, 6, 942072), True, datetime.datetime(2023, 1, 17, 4, 57, 6, 942072), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:57:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 57, 6, 942072), True, datetime.datetime(2023, 1, 17, 4, 57, 6, 942072), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 4, 57, 6, 942072), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 04:57:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:57:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:02:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:57:07,060 [231] [DEBUG] [app] Starting request: urn:request:bc64c8b1-4b76-41ce-b612-b040a2595f1e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:07,060 [236] [DEBUG] [app] Starting request: urn:request:2c109e74-6a03-4b06-8ae5-357ee4e463dd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:07,065 [239] [DEBUG] [app] Starting request: urn:request:0e656a80-444c-4d6a-bc20-0b7dbc2f9d57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:07,065 [230] [DEBUG] [app] Starting request: urn:request:29d9f04d-fbf8-438e-a00b-5318e90c5ca2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:07,065 [239] [DEBUG] [app] Ending request: urn:request:0e656a80-444c-4d6a-bc20-0b7dbc2f9d57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e656a80-444c-4d6a-bc20-0b7dbc2f9d57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 04:57:07,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:57:07,066 [230] [DEBUG] [app] Ending request: urn:request:29d9f04d-fbf8-438e-a00b-5318e90c5ca2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:29d9f04d-fbf8-438e-a00b-5318e90c5ca2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:57:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:07,070 [235] [DEBUG] [app] Starting request: urn:request:39dd93e4-d9ce-4c66-b818-9a166d0f5444 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:07,071 [233] [DEBUG] [app] Starting request: urn:request:9f018f45-3033-4ec0-8f94-788dd3a20e52 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:07,071 [235] [DEBUG] [app] Ending request: urn:request:39dd93e4-d9ce-4c66-b818-9a166d0f5444 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39dd93e4-d9ce-4c66-b818-9a166d0f5444', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:07,071 [233] [DEBUG] [app] Ending request: urn:request:9f018f45-3033-4ec0-8f94-788dd3a20e52 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f018f45-3033-4ec0-8f94-788dd3a20e52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:57:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:07,088 [231] [DEBUG] [app] Ending request: urn:request:bc64c8b1-4b76-41ce-b612-b040a2595f1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bc64c8b1-4b76-41ce-b612-b040a2595f1e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:07,088 [236] [DEBUG] [app] Ending request: urn:request:2c109e74-6a03-4b06-8ae5-357ee4e463dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2c109e74-6a03-4b06-8ae5-357ee4e463dd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:57:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) buildlogsarchiver stdout | 2023-01-17 04:57:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:57:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:40.738568+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 04:57:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:40 UTC)" (scheduled at 2023-01-17 04:57:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:57:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 57, 10, 739456), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:57:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:57:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:57:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:40 UTC)" executed successfully builder stdout | 2023-01-17 04:57:14,772 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:14,785 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:14,785 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:14,796 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:25.646354+00:00 (in 9.999559 seconds) notificationworker stdout | 2023-01-17 04:57:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:25 UTC)" (scheduled at 2023-01-17 04:57:15.646354+00:00) notificationworker stdout | 2023-01-17 04:57:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 15, 647215), True, datetime.datetime(2023, 1, 17, 4, 57, 15, 647215), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:19.537472+00:00 (in 2.600031 seconds) securityworker stdout | 2023-01-17 04:57:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:46 UTC)" (scheduled at 2023-01-17 04:57:16.936988+00:00) securityworker stdout | 2023-01-17 04:57:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:57:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:57:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:04.936835+00:00 (in 45.590269 seconds) repositorygcworker stdout | 2023-01-17 04:57:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:19 UTC)" (scheduled at 2023-01-17 04:57:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:57:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:57:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:46.936988+00:00 (in 27.399057 seconds) securityworker stdout | 2023-01-17 04:57:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:49 UTC)" (scheduled at 2023-01-17 04:57:19.537472+00:00) securityworker stdout | 2023-01-17 04:57:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:57:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:57:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:57:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:57:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:57:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:57:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:57:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:57:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:51.348605+00:00 (in 29.999589 seconds) gcworker stdout | 2023-01-17 04:57:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:51 UTC)" (scheduled at 2023-01-17 04:57:21.348605+00:00) gcworker stdout | 2023-01-17 04:57:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721841349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:57:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:57:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:57:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:57:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:57:22,060 [231] [DEBUG] [app] Starting request: urn:request:af678c53-f2de-4a2e-a818-09576bad0d1e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:22,060 [235] [DEBUG] [app] Starting request: urn:request:c85a8926-d5d7-478c-a79c-0e50065a45d1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:22,065 [243] [DEBUG] [app] Starting request: urn:request:e5c916bf-5a9f-4239-9ba5-a67798a791e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:22,065 [243] [DEBUG] [app] Ending request: urn:request:e5c916bf-5a9f-4239-9ba5-a67798a791e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5c916bf-5a9f-4239-9ba5-a67798a791e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:22,065 [230] [DEBUG] [app] Starting request: urn:request:516f2431-602d-431b-bfd3-8ec253a55682 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:57:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:57:22,066 [230] [DEBUG] [app] Ending request: urn:request:516f2431-602d-431b-bfd3-8ec253a55682 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:516f2431-602d-431b-bfd3-8ec253a55682', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:57:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:22,070 [236] [DEBUG] [app] Starting request: urn:request:ace5ab63-83b1-4a0b-99b1-531fc877b5a7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:22,070 [236] [DEBUG] [app] Ending request: urn:request:ace5ab63-83b1-4a0b-99b1-531fc877b5a7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ace5ab63-83b1-4a0b-99b1-531fc877b5a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:22,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:57:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:22,071 [233] [DEBUG] [app] Starting request: urn:request:53a94f4d-a07f-4f51-9681-9dd43e72ad79 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:22,072 [233] [DEBUG] [app] Ending request: urn:request:53a94f4d-a07f-4f51-9681-9dd43e72ad79 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53a94f4d-a07f-4f51-9681-9dd43e72ad79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:22,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:22,089 [231] [DEBUG] [app] Ending request: urn:request:af678c53-f2de-4a2e-a818-09576bad0d1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af678c53-f2de-4a2e-a818-09576bad0d1e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:22,089 [235] [DEBUG] [app] Ending request: urn:request:c85a8926-d5d7-478c-a79c-0e50065a45d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c85a8926-d5d7-478c-a79c-0e50065a45d1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:57:22,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) builder stdout | 2023-01-17 04:57:24,799 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:24,811 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:24,811 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:24,822 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:35.646354+00:00 (in 9.999580 seconds) notificationworker stdout | 2023-01-17 04:57:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:35 UTC)" (scheduled at 2023-01-17 04:57:25.646354+00:00) notificationworker stdout | 2023-01-17 04:57:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 25, 647167), True, datetime.datetime(2023, 1, 17, 4, 57, 25, 647167), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:57:27,069 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:57:27,102 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:57:27,975 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:57:28,521 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:57:28,603 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:57:28,871 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} exportactionlogsworker stdout | 2023-01-17 04:57:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:57:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:35.853396+00:00 (in 6.889225 seconds) exportactionlogsworker stdout | 2023-01-17 04:57:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:28 UTC)" (scheduled at 2023-01-17 04:57:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:57:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:57:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:28 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:29,035 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:57:30,560 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:57:30,563 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:57:30,566 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:57:30,568 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:57:30,570 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:57:32,023 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:57:32,398 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:57:33,574 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:57:33,577 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:57:33,816 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:57:34,368 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:57:34,495 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:57:34,570 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:57:34,572 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:57:34,574 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:57:34,750 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:57:34,779 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:57:34,781 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:57:34,783 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:57:34,785 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:57:34,788 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:57:34,793 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:57:34,796 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:57:34,800 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:57:34,803 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} builder stdout | 2023-01-17 04:57:34,832 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:34,845 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:34,845 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:34,855 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 04:57:34,957 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 04:57:35,286 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:57:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:45.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 04:57:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:45 UTC)" (scheduled at 2023-01-17 04:57:35.646354+00:00) notificationworker stdout | 2023-01-17 04:57:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 35, 647215), True, datetime.datetime(2023, 1, 17, 4, 57, 35, 647215), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:35,667 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:35,667 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:35,668 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:45 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:57:35,780 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:57:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:57:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:28.963662+00:00 (in 53.109765 seconds) exportactionlogsworker stdout | 2023-01-17 04:57:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:35 UTC)" (scheduled at 2023-01-17 04:57:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:57:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:57:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 35, 854291), True, datetime.datetime(2023, 1, 17, 4, 57, 35, 854291), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:57:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:57:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:57:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:57:36,010 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:57:36,043 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:57:37,060 [231] [DEBUG] [app] Starting request: urn:request:0914381d-4b0c-4c33-9bd2-a0178dc27a8a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:37,060 [236] [DEBUG] [app] Starting request: urn:request:0b24e826-dbab-401e-93af-0a610b01b6c6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:37,065 [243] [DEBUG] [app] Starting request: urn:request:12e02c7c-f97f-4f22-951a-e952efba7797 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:37,065 [230] [DEBUG] [app] Starting request: urn:request:5a083c3c-5fd6-4e90-b521-9f2ecfbb23bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:37,065 [243] [DEBUG] [app] Ending request: urn:request:12e02c7c-f97f-4f22-951a-e952efba7797 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:12e02c7c-f97f-4f22-951a-e952efba7797', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:57:37,066 [230] [DEBUG] [app] Ending request: urn:request:5a083c3c-5fd6-4e90-b521-9f2ecfbb23bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a083c3c-5fd6-4e90-b521-9f2ecfbb23bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:57:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:57:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:37,070 [231] [DEBUG] [app] Starting request: urn:request:f3338d08-002e-473b-9370-44bbf1c335ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:37,070 [231] [DEBUG] [app] Ending request: urn:request:f3338d08-002e-473b-9370-44bbf1c335ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3338d08-002e-473b-9370-44bbf1c335ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:57:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:37,072 [235] [DEBUG] [app] Starting request: urn:request:18c74fd4-df58-44ec-82a1-541e388afd3b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:37,073 [235] [DEBUG] [app] Ending request: urn:request:18c74fd4-df58-44ec-82a1-541e388afd3b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18c74fd4-df58-44ec-82a1-541e388afd3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:37,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:37,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:37,074 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:37,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:37,087 [236] [DEBUG] [app] Ending request: urn:request:0b24e826-dbab-401e-93af-0a610b01b6c6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b24e826-dbab-401e-93af-0a610b01b6c6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 04:57:37,089 [231] [DEBUG] [app] Ending request: urn:request:0914381d-4b0c-4c33-9bd2-a0178dc27a8a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0914381d-4b0c-4c33-9bd2-a0178dc27a8a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:37,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:37,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.031) securityscanningnotificationworker stdout | 2023-01-17 04:57:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:57:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:37.537407+00:00 (in 0.280352 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:57:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:37 UTC)" (scheduled at 2023-01-17 04:57:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:57:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:57:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 37, 257459), True, datetime.datetime(2023, 1, 17, 4, 57, 37, 257459), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:57:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:57:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:57:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:57:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:57:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:37.256652+00:00 (in 59.718862 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:57:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:37 UTC)" (scheduled at 2023-01-17 04:57:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:57:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:57:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:57:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:57:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:10.738568+00:00 (in 29.999563 seconds) buildlogsarchiver stdout | 2023-01-17 04:57:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:10 UTC)" (scheduled at 2023-01-17 04:57:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:57:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 57, 40, 739353), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:57:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:57:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:57:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:57:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:57:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:42.941536+00:00 (in 1.792802 seconds) namespacegcworker stdout | 2023-01-17 04:57:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:41 UTC)" (scheduled at 2023-01-17 04:57:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:57:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:57:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 41, 149162), True, datetime.datetime(2023, 1, 17, 4, 57, 41, 149162), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:57:41,162 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:57:41,162 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:57:41,162 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:57:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:57:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:41.148245+00:00 (in 58.206198 seconds) namespacegcworker stdout | 2023-01-17 04:57:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:42 UTC)" (scheduled at 2023-01-17 04:57:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:57:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:57:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:58:42 UTC)" executed successfully builder stdout | 2023-01-17 04:57:44,866 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:44,878 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:44,879 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:44,889 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:55.646354+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 04:57:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:55 UTC)" (scheduled at 2023-01-17 04:57:45.646354+00:00) notificationworker stdout | 2023-01-17 04:57:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 45, 647241), True, datetime.datetime(2023, 1, 17, 4, 57, 45, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:57:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:57:49.537472+00:00 (in 2.600003 seconds) securityworker stdout | 2023-01-17 04:57:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:16 UTC)" (scheduled at 2023-01-17 04:57:46.936988+00:00) securityworker stdout | 2023-01-17 04:57:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:46,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:57:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:57:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:16.936988+00:00 (in 27.399044 seconds) securityworker stdout | 2023-01-17 04:57:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:19 UTC)" (scheduled at 2023-01-17 04:57:49.537472+00:00) securityworker stdout | 2023-01-17 04:57:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:57:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:57:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:57:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:57:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:57:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:57:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:57:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:57:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:57:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:57:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:57:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:57:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:57:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:57:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:57:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:21.348605+00:00 (in 29.999554 seconds) gcworker stdout | 2023-01-17 04:57:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:21 UTC)" (scheduled at 2023-01-17 04:57:51.348605+00:00) gcworker stdout | 2023-01-17 04:57:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721871349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:57:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:57:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:57:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:57:52,059 [233] [DEBUG] [app] Starting request: urn:request:f34e85aa-df74-417e-a3bc-6837ee927721 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:52,060 [231] [DEBUG] [app] Starting request: urn:request:2774e523-c05e-4f21-90cf-87cfff13520d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:57:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:57:52,065 [243] [DEBUG] [app] Starting request: urn:request:bdbf0aef-10c0-4424-9710-1a0276fc92c0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:57:52,065 [243] [DEBUG] [app] Ending request: urn:request:bdbf0aef-10c0-4424-9710-1a0276fc92c0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bdbf0aef-10c0-4424-9710-1a0276fc92c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:52,065 [241] [DEBUG] [app] Starting request: urn:request:2b3a1e2b-3f84-451a-841f-51f7854d32d0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:57:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:57:52,065 [241] [DEBUG] [app] Ending request: urn:request:2b3a1e2b-3f84-451a-841f-51f7854d32d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2b3a1e2b-3f84-451a-841f-51f7854d32d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:57:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:57:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:57:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:57:52,070 [235] [DEBUG] [app] Starting request: urn:request:3595d101-e320-404b-bcd0-391e1ebdc294 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:52,071 [235] [DEBUG] [app] Ending request: urn:request:3595d101-e320-404b-bcd0-391e1ebdc294 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3595d101-e320-404b-bcd0-391e1ebdc294', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:57:52,071 [236] [DEBUG] [app] Starting request: urn:request:343190d1-6954-4037-923a-0b959006b943 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:57:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:52,072 [236] [DEBUG] [app] Ending request: urn:request:343190d1-6954-4037-923a-0b959006b943 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:343190d1-6954-4037-923a-0b959006b943', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:57:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:57:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:57:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:57:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:57:52,088 [233] [DEBUG] [app] Ending request: urn:request:f34e85aa-df74-417e-a3bc-6837ee927721 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f34e85aa-df74-417e-a3bc-6837ee927721', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:57:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:57:52,088 [231] [DEBUG] [app] Ending request: urn:request:2774e523-c05e-4f21-90cf-87cfff13520d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2774e523-c05e-4f21-90cf-87cfff13520d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 04:57:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:57:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 04:57:54,899 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:57:54,912 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:57:54,913 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:57:54,923 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:57:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:01.636362+00:00 (in 5.989527 seconds) notificationworker stdout | 2023-01-17 04:57:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:05 UTC)" (scheduled at 2023-01-17 04:57:55.646354+00:00) notificationworker stdout | 2023-01-17 04:57:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:57:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 57, 55, 647256), True, datetime.datetime(2023, 1, 17, 4, 57, 55, 647256), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:57:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:57:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:57:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:57:57,105 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:57:57,136 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:57:58,013 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:57:58,543 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:57:58,638 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:57:58,904 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:57:59,061 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:58:00,567 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:58:00,570 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:58:00,572 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:58:00,574 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:58:00,577 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:58:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:05.646354+00:00 (in 4.009492 seconds) notificationworker stdout | 2023-01-17 04:58:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:01 UTC)" (scheduled at 2023-01-17 04:58:01.636362+00:00) notificationworker stdout | 2023-01-17 04:58:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:58:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:58:02,049 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:58:02,433 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:58:03,606 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:58:03,609 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:58:03,831 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:58:04,381 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:58:04,531 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:58:04,577 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:58:04,579 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:58:04,582 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} securityscanningnotificationworker stdout | 2023-01-17 04:58:04,785 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-registry stdout | 2023-01-17 04:58:04,790 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:58:04,792 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:58:04,795 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:58:04,797 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:58:04,800 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:58:04,803 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:58:04,806 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:58:04,808 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:58:04,811 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} builder stdout | 2023-01-17 04:58:04,933 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue repositorygcworker stdout | 2023-01-17 04:58:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:58:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:19.346120+00:00 (in 14.408857 seconds) repositorygcworker stdout | 2023-01-17 04:58:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:04 UTC)" (scheduled at 2023-01-17 04:58:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:58:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:58:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 4, 937675), True, datetime.datetime(2023, 1, 17, 4, 58, 4, 937675), 0, 'repositorygc/%', 50, 1, 0]) builder stdout | 2023-01-17 04:58:04,946 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:04,946 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics repositorygcworker stdout | 2023-01-17 04:58:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:58:04,948 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:58:04,948 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:04 UTC)" executed successfully builder stdout | 2023-01-17 04:58:04,957 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 04:58:04,978 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 04:58:05,312 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:58:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:15.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 04:58:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:15 UTC)" (scheduled at 2023-01-17 04:58:05.646354+00:00) notificationworker stdout | 2023-01-17 04:58:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 5, 647235), True, datetime.datetime(2023, 1, 17, 4, 58, 5, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:15 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:58:05,812 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:58:06,045 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:58:06,078 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:58:07,060 [236] [DEBUG] [app] Starting request: urn:request:2e09e4c0-d461-42e2-88de-5247a73bccb7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:07,060 [233] [DEBUG] [app] Starting request: urn:request:96517c0c-abf4-4e86-a929-9349464177d3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:07,065 [241] [DEBUG] [app] Starting request: urn:request:279824f2-3e12-4138-ab9c-18e99cb7fbac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:07,065 [242] [DEBUG] [app] Starting request: urn:request:1277278c-7284-4104-a4f0-ca0e9caebbb6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:07,065 [241] [DEBUG] [app] Ending request: urn:request:279824f2-3e12-4138-ab9c-18e99cb7fbac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:279824f2-3e12-4138-ab9c-18e99cb7fbac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:58:07,065 [242] [DEBUG] [app] Ending request: urn:request:1277278c-7284-4104-a4f0-ca0e9caebbb6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1277278c-7284-4104-a4f0-ca0e9caebbb6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:07,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:07,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:58:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:58:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:07,070 [236] [DEBUG] [app] Starting request: urn:request:38d7a3a8-d7b1-434d-8c0a-10c04eacb1ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:07,069 [231] [DEBUG] [app] Starting request: urn:request:148157a7-2557-4fb8-bbc8-7f7e6a97998d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:07,070 [236] [DEBUG] [app] Ending request: urn:request:38d7a3a8-d7b1-434d-8c0a-10c04eacb1ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:38d7a3a8-d7b1-434d-8c0a-10c04eacb1ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:07,070 [231] [DEBUG] [app] Ending request: urn:request:148157a7-2557-4fb8-bbc8-7f7e6a97998d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:148157a7-2557-4fb8-bbc8-7f7e6a97998d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:58:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:07,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:07,084 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:07,088 [233] [DEBUG] [app] Ending request: urn:request:96517c0c-abf4-4e86-a929-9349464177d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:96517c0c-abf4-4e86-a929-9349464177d3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:07,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 04:58:07,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:58:07,094 [236] [DEBUG] [app] Ending request: urn:request:2e09e4c0-d461-42e2-88de-5247a73bccb7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e09e4c0-d461-42e2-88de-5247a73bccb7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:07,094 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:07,094 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.036) buildlogsarchiver stdout | 2023-01-17 04:58:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:58:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:40.738568+00:00 (in 29.999594 seconds) buildlogsarchiver stdout | 2023-01-17 04:58:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:40 UTC)" (scheduled at 2023-01-17 04:58:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:58:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 58, 10, 739373), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:58:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:58:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:58:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:40 UTC)" executed successfully builder stdout | 2023-01-17 04:58:14,963 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:14,976 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:14,976 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:14,987 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:25.646354+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 04:58:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:25 UTC)" (scheduled at 2023-01-17 04:58:15.646354+00:00) notificationworker stdout | 2023-01-17 04:58:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 15, 647260), True, datetime.datetime(2023, 1, 17, 4, 58, 15, 647260), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:19.537472+00:00 (in 2.600026 seconds) securityworker stdout | 2023-01-17 04:58:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:46 UTC)" (scheduled at 2023-01-17 04:58:16.936988+00:00) securityworker stdout | 2023-01-17 04:58:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:58:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:58:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:04.936835+00:00 (in 45.590285 seconds) repositorygcworker stdout | 2023-01-17 04:58:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:19 UTC)" (scheduled at 2023-01-17 04:58:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:58:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:58:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:46.936988+00:00 (in 27.398998 seconds) securityworker stdout | 2023-01-17 04:58:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:49 UTC)" (scheduled at 2023-01-17 04:58:19.537472+00:00) securityworker stdout | 2023-01-17 04:58:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:58:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:58:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:19,558 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:19,558 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:58:19,558 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:58:19,558 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:58:19,559 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:19,560 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:19,560 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:19,560 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:58:19,560 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:58:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:58:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:51.348605+00:00 (in 29.999552 seconds) gcworker stdout | 2023-01-17 04:58:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:51 UTC)" (scheduled at 2023-01-17 04:58:21.348605+00:00) gcworker stdout | 2023-01-17 04:58:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721901349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:58:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:58:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:58:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:58:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:58:22,060 [231] [DEBUG] [app] Starting request: urn:request:326fac11-ec53-41ee-bbe8-5f17c5d553d1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:22,060 [235] [DEBUG] [app] Starting request: urn:request:a7c974cc-8c25-41e0-998e-9f78aab071f9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:22,065 [230] [DEBUG] [app] Starting request: urn:request:d24ad713-8995-4216-8b75-149ccaab382f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:22,066 [230] [DEBUG] [app] Ending request: urn:request:d24ad713-8995-4216-8b75-149ccaab382f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d24ad713-8995-4216-8b75-149ccaab382f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:58:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:58:22,066 [232] [DEBUG] [app] Starting request: urn:request:eb5afe10-5601-4f21-a9e0-2c4ca1d7020f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:58:22,067 [232] [DEBUG] [app] Ending request: urn:request:eb5afe10-5601-4f21-a9e0-2c4ca1d7020f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb5afe10-5601-4f21-a9e0-2c4ca1d7020f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-registry stdout | 2023-01-17 04:58:22,068 [232] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:22,068 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:22,069 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:22,070 [233] [DEBUG] [app] Starting request: urn:request:7f5e05ed-98a6-48fe-8564-898b4dbf3007 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:22,071 [233] [DEBUG] [app] Ending request: urn:request:7f5e05ed-98a6-48fe-8564-898b4dbf3007 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f5e05ed-98a6-48fe-8564-898b4dbf3007', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:22,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 04:58:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:22,072 [236] [DEBUG] [app] Starting request: urn:request:2645ae14-3966-42b7-9fa0-47efc8856270 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:22,072 [236] [DEBUG] [app] Ending request: urn:request:2645ae14-3966-42b7-9fa0-47efc8856270 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2645ae14-3966-42b7-9fa0-47efc8856270', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:22,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:22,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:22,088 [235] [DEBUG] [app] Ending request: urn:request:a7c974cc-8c25-41e0-998e-9f78aab071f9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7c974cc-8c25-41e0-998e-9f78aab071f9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:22,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 04:58:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:58:22,089 [231] [DEBUG] [app] Ending request: urn:request:326fac11-ec53-41ee-bbe8-5f17c5d553d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:326fac11-ec53-41ee-bbe8-5f17c5d553d1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 04:58:24,990 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:25,002 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:25,002 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:25,013 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:35.646354+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 04:58:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:35 UTC)" (scheduled at 2023-01-17 04:58:25.646354+00:00) notificationworker stdout | 2023-01-17 04:58:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 25, 647254), True, datetime.datetime(2023, 1, 17, 4, 58, 25, 647254), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:25,665 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:25,665 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:25,665 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:58:27,141 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:58:27,157 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:58:28,027 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:58:28,551 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:58:28,650 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:58:28,932 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} exportactionlogsworker stdout | 2023-01-17 04:58:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:58:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:35.853396+00:00 (in 6.889302 seconds) exportactionlogsworker stdout | 2023-01-17 04:58:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:28 UTC)" (scheduled at 2023-01-17 04:58:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:58:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:58:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:28 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:29,090 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:58:30,575 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:58:30,578 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:58:30,580 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:58:30,582 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:58:30,585 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:58:32,070 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:58:32,453 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 04:58:33,612 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 04:58:33,615 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 04:58:33,870 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:58:34,388 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 04:58:34,566 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-secscan stdout | 2023-01-17 04:58:34,584 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:58:34,586 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:58:34,588 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 04:58:34,800 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:58:34,803 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:58:34,805 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:58:34,808 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:58:34,810 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:58:34,812 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:58:34,816 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:58:34,818 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:58:34,820 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} securityscanningnotificationworker stdout | 2023-01-17 04:58:34,822 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 04:58:34,989 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 04:58:35,020 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:35,033 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:35,033 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:35,044 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 04:58:35,322 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:58:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:36.635997+00:00 (in 0.989158 seconds) notificationworker stdout | 2023-01-17 04:58:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:45 UTC)" (scheduled at 2023-01-17 04:58:35.646354+00:00) notificationworker stdout | 2023-01-17 04:58:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 35, 647265), True, datetime.datetime(2023, 1, 17, 4, 58, 35, 647265), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:45 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:58:35,823 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:58:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:58:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:28.963662+00:00 (in 53.109782 seconds) exportactionlogsworker stdout | 2023-01-17 04:58:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:35 UTC)" (scheduled at 2023-01-17 04:58:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:58:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:58:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 35, 854279), True, datetime.datetime(2023, 1, 17, 4, 58, 35, 854279), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:58:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:58:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:58:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:58:36,056 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:58:36,113 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 04:58:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:45.646354+00:00 (in 9.009873 seconds) notificationworker stdout | 2023-01-17 04:58:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:03:36 UTC)" (scheduled at 2023-01-17 04:58:36.635997+00:00) notificationworker stdout | 2023-01-17 04:58:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 4, 58, 36, 636844), 'notification/%']) notificationworker stdout | 2023-01-17 04:58:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 58, 36, 636844), True, datetime.datetime(2023, 1, 17, 4, 58, 36, 636844), 0, 'notification/%']) notificationworker stdout | 2023-01-17 04:58:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 4, 58, 36, 636844), True, datetime.datetime(2023, 1, 17, 4, 58, 36, 636844), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 4, 58, 36, 636844), 'notification/%']) notificationworker stdout | 2023-01-17 04:58:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:03:36 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:58:37,059 [231] [DEBUG] [app] Starting request: urn:request:8baeafb4-c3e0-45ce-a962-9c88f0fcc2ef (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:37,059 [235] [DEBUG] [app] Starting request: urn:request:4d0b6201-0243-4004-9f14-1605497de026 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:37,065 [230] [DEBUG] [app] Starting request: urn:request:b6803c76-2b77-4fca-853a-0ccfcaad48e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:37,065 [243] [DEBUG] [app] Starting request: urn:request:1f8d990c-4c50-442b-a672-c919dc1a0d9e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:37,065 [230] [DEBUG] [app] Ending request: urn:request:b6803c76-2b77-4fca-853a-0ccfcaad48e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b6803c76-2b77-4fca-853a-0ccfcaad48e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:37,066 [243] [DEBUG] [app] Ending request: urn:request:1f8d990c-4c50-442b-a672-c919dc1a0d9e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1f8d990c-4c50-442b-a672-c919dc1a0d9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:58:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:58:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:58:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:37,070 [236] [DEBUG] [app] Starting request: urn:request:d2ea073f-f63c-447c-bb7b-2b8ad570e8b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:37,070 [231] [DEBUG] [app] Starting request: urn:request:96da9dc8-dad7-4875-8dac-7d79c9738bf5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:37,071 [236] [DEBUG] [app] Ending request: urn:request:d2ea073f-f63c-447c-bb7b-2b8ad570e8b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d2ea073f-f63c-447c-bb7b-2b8ad570e8b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:37,071 [231] [DEBUG] [app] Ending request: urn:request:96da9dc8-dad7-4875-8dac-7d79c9738bf5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:96da9dc8-dad7-4875-8dac-7d79c9738bf5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:37,088 [235] [DEBUG] [app] Ending request: urn:request:4d0b6201-0243-4004-9f14-1605497de026 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d0b6201-0243-4004-9f14-1605497de026', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 04:58:37,097 [231] [DEBUG] [app] Ending request: urn:request:8baeafb4-c3e0-45ce-a962-9c88f0fcc2ef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8baeafb4-c3e0-45ce-a962-9c88f0fcc2ef', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:37,097 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:58:37,097 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.039 47 0.039) securityscanningnotificationworker stdout | 2023-01-17 04:58:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:58:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:37.537407+00:00 (in 0.280349 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:58:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:37 UTC)" (scheduled at 2023-01-17 04:58:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:58:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:58:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 37, 257505), True, datetime.datetime(2023, 1, 17, 4, 58, 37, 257505), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:58:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:58:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:58:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:58:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:58:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:37.256652+00:00 (in 59.718818 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:58:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:37 UTC)" (scheduled at 2023-01-17 04:58:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:58:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:58:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:58:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:58:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:10.738568+00:00 (in 29.999564 seconds) buildlogsarchiver stdout | 2023-01-17 04:58:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:10 UTC)" (scheduled at 2023-01-17 04:58:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:58:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 58, 40, 739375), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:58:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:58:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:58:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:58:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:58:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:42.941536+00:00 (in 1.792831 seconds) namespacegcworker stdout | 2023-01-17 04:58:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:41 UTC)" (scheduled at 2023-01-17 04:58:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:58:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:58:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 41, 149107), True, datetime.datetime(2023, 1, 17, 4, 58, 41, 149107), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:58:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:58:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:58:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:58:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:58:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:41.148245+00:00 (in 58.206233 seconds) namespacegcworker stdout | 2023-01-17 04:58:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:42 UTC)" (scheduled at 2023-01-17 04:58:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:58:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:58:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 04:59:42 UTC)" executed successfully builder stdout | 2023-01-17 04:58:45,054 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:45,067 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:45,067 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:45,078 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:55.646354+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 04:58:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:55 UTC)" (scheduled at 2023-01-17 04:58:45.646354+00:00) notificationworker stdout | 2023-01-17 04:58:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 45, 647260), True, datetime.datetime(2023, 1, 17, 4, 58, 45, 647260), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:58:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:58:49.537472+00:00 (in 2.599981 seconds) securityworker stdout | 2023-01-17 04:58:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:16 UTC)" (scheduled at 2023-01-17 04:58:46.936988+00:00) securityworker stdout | 2023-01-17 04:58:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:58:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:58:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:16.936988+00:00 (in 27.399014 seconds) securityworker stdout | 2023-01-17 04:58:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:19 UTC)" (scheduled at 2023-01-17 04:58:49.537472+00:00) securityworker stdout | 2023-01-17 04:58:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:58:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:58:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:58:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:58:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:58:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:58:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:58:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:58:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:58:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:58:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:58:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:58:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:58:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:58:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:58:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:21.348605+00:00 (in 29.999550 seconds) gcworker stdout | 2023-01-17 04:58:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:21 UTC)" (scheduled at 2023-01-17 04:58:51.348605+00:00) gcworker stdout | 2023-01-17 04:58:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721931349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:58:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:58:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:58:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:58:52,059 [231] [DEBUG] [app] Starting request: urn:request:80da65f9-9e37-4723-9e0f-360e4243d9e8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:52,059 [236] [DEBUG] [app] Starting request: urn:request:078e0573-3f3a-4036-90eb-09141f679308 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:58:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:58:52,064 [242] [DEBUG] [app] Starting request: urn:request:7bab86a7-abe6-4b56-a9df-620e43b0b719 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:52,065 [243] [DEBUG] [app] Starting request: urn:request:506fcb80-fd87-400f-913e-6c6ea844aca6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:58:52,065 [242] [DEBUG] [app] Ending request: urn:request:7bab86a7-abe6-4b56-a9df-620e43b0b719 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7bab86a7-abe6-4b56-a9df-620e43b0b719', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 04:58:52,065 [243] [DEBUG] [app] Ending request: urn:request:506fcb80-fd87-400f-913e-6c6ea844aca6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:506fcb80-fd87-400f-913e-6c6ea844aca6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:58:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:58:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:58:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:58:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:52,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:58:52,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:52,069 [231] [DEBUG] [app] Starting request: urn:request:e17a2afc-95f2-4f2f-8ea8-69dc54b944e9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:52,069 [231] [DEBUG] [app] Ending request: urn:request:e17a2afc-95f2-4f2f-8ea8-69dc54b944e9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e17a2afc-95f2-4f2f-8ea8-69dc54b944e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:52,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:52,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:58:52,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:52,071 [233] [DEBUG] [app] Starting request: urn:request:8f724306-44dd-4c3e-8182-c5b9a8c314ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:58:52,072 [233] [DEBUG] [app] Ending request: urn:request:8f724306-44dd-4c3e-8182-c5b9a8c314ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8f724306-44dd-4c3e-8182-c5b9a8c314ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:58:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:58:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:58:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:58:52,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:58:52,087 [236] [DEBUG] [app] Ending request: urn:request:078e0573-3f3a-4036-90eb-09141f679308 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:078e0573-3f3a-4036-90eb-09141f679308', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:52,087 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 04:58:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:58:52,088 [231] [DEBUG] [app] Ending request: urn:request:80da65f9-9e37-4723-9e0f-360e4243d9e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:80da65f9-9e37-4723-9e0f-360e4243d9e8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:58:52,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 04:58:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:58:55,088 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:58:55,102 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:58:55,102 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:58:55,112 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:58:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:01.636362+00:00 (in 5.989517 seconds) notificationworker stdout | 2023-01-17 04:58:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:05 UTC)" (scheduled at 2023-01-17 04:58:55.646354+00:00) notificationworker stdout | 2023-01-17 04:58:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:58:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 58, 55, 647285), True, datetime.datetime(2023, 1, 17, 4, 58, 55, 647285), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:58:55,669 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:58:55,669 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:58:55,670 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:05 UTC)" executed successfully gcworker stdout | 2023-01-17 04:58:57,148 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:58:57,166 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:58:58,050 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:58:58,583 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:58:58,664 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:58:58,950 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:58:59,100 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:59:00,581 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:59:00,584 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:59:00,587 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:59:00,589 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:59:00,591 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 04:59:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:05.646354+00:00 (in 4.009522 seconds) notificationworker stdout | 2023-01-17 04:59:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:01 UTC)" (scheduled at 2023-01-17 04:59:01.636362+00:00) notificationworker stdout | 2023-01-17 04:59:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 04:59:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 04:59:02,101 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:59:02,486 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:59:03,623 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:59:03,647 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:59:03,902 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:59:04,402 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 04:59:04,591 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:59:04,593 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:59:04,596 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 04:59:04,601 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 04:59:04,807 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:59:04,813 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:59:04,815 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:59:04,820 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:59:04,822 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:59:04,824 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:59:04,826 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:59:04,829 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:59:04,831 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} securityscanningnotificationworker stdout | 2023-01-17 04:59:04,855 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositorygcworker stdout | 2023-01-17 04:59:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:59:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:19.346120+00:00 (in 14.408846 seconds) repositorygcworker stdout | 2023-01-17 04:59:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:04 UTC)" (scheduled at 2023-01-17 04:59:04.936835+00:00) repositorygcworker stdout | 2023-01-17 04:59:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 04:59:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 4, 937692), True, datetime.datetime(2023, 1, 17, 4, 59, 4, 937692), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 04:59:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 04:59:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 04:59:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:04 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 04:59:05,025 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 04:59:05,121 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:05,133 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:05,133 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:05,144 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 04:59:05,346 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:59:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:15.646354+00:00 (in 9.999558 seconds) notificationworker stdout | 2023-01-17 04:59:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:15 UTC)" (scheduled at 2023-01-17 04:59:05.646354+00:00) notificationworker stdout | 2023-01-17 04:59:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 5, 647179), True, datetime.datetime(2023, 1, 17, 4, 59, 5, 647179), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:15 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:59:05,858 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:59:06,070 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:59:06,135 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:59:07,060 [231] [DEBUG] [app] Starting request: urn:request:6cdcc9c5-2742-4b15-b149-34f10c285992 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:07,061 [235] [DEBUG] [app] Starting request: urn:request:087501a3-1282-4d01-a34e-a500662ce9ec (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:07,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:07,065 [230] [DEBUG] [app] Starting request: urn:request:7f11778f-69be-4c89-b855-65aa36aec4f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,066 [243] [DEBUG] [app] Starting request: urn:request:8542527e-3601-4cfe-95f8-38f452330553 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,066 [230] [DEBUG] [app] Ending request: urn:request:7f11778f-69be-4c89-b855-65aa36aec4f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f11778f-69be-4c89-b855-65aa36aec4f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,066 [243] [DEBUG] [app] Ending request: urn:request:8542527e-3601-4cfe-95f8-38f452330553 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8542527e-3601-4cfe-95f8-38f452330553', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 04:59:07,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:59:07,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:07,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:07,072 [236] [DEBUG] [app] Starting request: urn:request:a714fba5-6465-480e-b7f9-afa58329f736 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:07,072 [236] [DEBUG] [app] Ending request: urn:request:a714fba5-6465-480e-b7f9-afa58329f736 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a714fba5-6465-480e-b7f9-afa58329f736', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:07,072 [233] [DEBUG] [app] Starting request: urn:request:c94e82d4-c3cd-46e6-9db8-7b685b085f0d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:07,073 [233] [DEBUG] [app] Ending request: urn:request:c94e82d4-c3cd-46e6-9db8-7b685b085f0d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c94e82d4-c3cd-46e6-9db8-7b685b085f0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:07,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:07,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:07,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:07,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:07,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:07,089 [235] [DEBUG] [app] Ending request: urn:request:087501a3-1282-4d01-a34e-a500662ce9ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:087501a3-1282-4d01-a34e-a500662ce9ec', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:07,090 [231] [DEBUG] [app] Ending request: urn:request:6cdcc9c5-2742-4b15-b149-34f10c285992 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cdcc9c5-2742-4b15-b149-34f10c285992', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:07,090 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:07,090 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 04:59:07,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:59:07,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 04:59:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:59:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:40.738568+00:00 (in 29.999539 seconds) buildlogsarchiver stdout | 2023-01-17 04:59:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:40 UTC)" (scheduled at 2023-01-17 04:59:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:59:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 59, 10, 739434), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:59:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:59:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:59:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:40 UTC)" executed successfully builder stdout | 2023-01-17 04:59:15,144 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:15,157 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:15,157 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:15,168 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:25.646354+00:00 (in 9.999562 seconds) notificationworker stdout | 2023-01-17 04:59:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:25 UTC)" (scheduled at 2023-01-17 04:59:15.646354+00:00) notificationworker stdout | 2023-01-17 04:59:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 15, 647200), True, datetime.datetime(2023, 1, 17, 4, 59, 15, 647200), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:25 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:19.537472+00:00 (in 2.600031 seconds) securityworker stdout | 2023-01-17 04:59:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:46 UTC)" (scheduled at 2023-01-17 04:59:16.936988+00:00) securityworker stdout | 2023-01-17 04:59:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 04:59:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 04:59:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:04.936835+00:00 (in 45.590253 seconds) repositorygcworker stdout | 2023-01-17 04:59:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:19 UTC)" (scheduled at 2023-01-17 04:59:19.346120+00:00) repositorygcworker stdout | 2023-01-17 04:59:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 04:59:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:19 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:46.936988+00:00 (in 27.399096 seconds) securityworker stdout | 2023-01-17 04:59:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:49 UTC)" (scheduled at 2023-01-17 04:59:19.537472+00:00) securityworker stdout | 2023-01-17 04:59:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:59:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:59:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:19,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:59:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:59:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:59:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:59:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:49 UTC)" executed successfully gcworker stdout | 2023-01-17 04:59:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:59:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:51.348605+00:00 (in 29.999560 seconds) gcworker stdout | 2023-01-17 04:59:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:51 UTC)" (scheduled at 2023-01-17 04:59:21.348605+00:00) gcworker stdout | 2023-01-17 04:59:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721961349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:59:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:59:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:59:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 04:59:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:59:22,060 [231] [DEBUG] [app] Starting request: urn:request:19b88cde-8fc4-4d7a-b748-1d99e4ab449b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:22,060 [236] [DEBUG] [app] Starting request: urn:request:e9a9b4e0-75c8-4c9a-9b19-6ad88cac315a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:22,065 [241] [DEBUG] [app] Starting request: urn:request:e6e75b05-ae1d-407b-84fd-9a0bbaee5c5c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:22,065 [230] [DEBUG] [app] Starting request: urn:request:7ef082eb-f210-49b0-9443-9bbbc692a778 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:22,065 [241] [DEBUG] [app] Ending request: urn:request:e6e75b05-ae1d-407b-84fd-9a0bbaee5c5c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e6e75b05-ae1d-407b-84fd-9a0bbaee5c5c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:22,065 [230] [DEBUG] [app] Ending request: urn:request:7ef082eb-f210-49b0-9443-9bbbc692a778 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ef082eb-f210-49b0-9443-9bbbc692a778', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:22,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 04:59:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:22,070 [235] [DEBUG] [app] Starting request: urn:request:7ad697d5-faa5-494e-af70-865816577137 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:22,070 [236] [DEBUG] [app] Starting request: urn:request:2e5de4ca-4204-46af-a270-21057bd45259 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:22,070 [235] [DEBUG] [app] Ending request: urn:request:7ad697d5-faa5-494e-af70-865816577137 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ad697d5-faa5-494e-af70-865816577137', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:22,071 [236] [DEBUG] [app] Ending request: urn:request:2e5de4ca-4204-46af-a270-21057bd45259 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e5de4ca-4204-46af-a270-21057bd45259', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:22,087 [231] [DEBUG] [app] Ending request: urn:request:19b88cde-8fc4-4d7a-b748-1d99e4ab449b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:19b88cde-8fc4-4d7a-b748-1d99e4ab449b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:22,088 [236] [DEBUG] [app] Ending request: urn:request:e9a9b4e0-75c8-4c9a-9b19-6ad88cac315a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e9a9b4e0-75c8-4c9a-9b19-6ad88cac315a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 04:59:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 04:59:25,178 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:25,191 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:25,191 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:25,201 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:35.646354+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 04:59:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:35 UTC)" (scheduled at 2023-01-17 04:59:25.646354+00:00) notificationworker stdout | 2023-01-17 04:59:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 25, 647200), True, datetime.datetime(2023, 1, 17, 4, 59, 25, 647200), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:35 UTC)" executed successfully gcworker stdout | 2023-01-17 04:59:27,183 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 04:59:27,201 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 04:59:28,086 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:59:28,599 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:59:28,689 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} exportactionlogsworker stdout | 2023-01-17 04:59:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:59:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:35.853396+00:00 (in 6.889238 seconds) exportactionlogsworker stdout | 2023-01-17 04:59:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:28 UTC)" (scheduled at 2023-01-17 04:59:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 04:59:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 04:59:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:28 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:59:28,986 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:59:29,135 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 04:59:30,589 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 04:59:30,592 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 04:59:30,595 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 04:59:30,597 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 04:59:30,599 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 04:59:32,133 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 04:59:32,522 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 04:59:33,648 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 04:59:33,682 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 04:59:33,908 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 04:59:34,423 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 04:59:34,597 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 04:59:34,600 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 04:59:34,602 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 04:59:34,627 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 04:59:34,816 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 04:59:34,821 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 04:59:34,824 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 04:59:34,829 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 04:59:34,831 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 04:59:34,834 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 04:59:34,836 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 04:59:34,839 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 04:59:34,841 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} securityscanningnotificationworker stdout | 2023-01-17 04:59:34,890 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 04:59:35,059 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 04:59:35,211 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:35,224 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:35,224 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:35,235 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 04:59:35,382 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 04:59:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:45.646354+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 04:59:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:45 UTC)" (scheduled at 2023-01-17 04:59:35.646354+00:00) notificationworker stdout | 2023-01-17 04:59:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 35, 647282), True, datetime.datetime(2023, 1, 17, 4, 59, 35, 647282), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 04:59:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 04:59:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:28.963662+00:00 (in 53.109748 seconds) exportactionlogsworker stdout | 2023-01-17 04:59:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:35 UTC)" (scheduled at 2023-01-17 04:59:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 04:59:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 04:59:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 35, 854397), True, datetime.datetime(2023, 1, 17, 4, 59, 35, 854397), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 04:59:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 04:59:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 04:59:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 04:59:35,893 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 04:59:36,085 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 04:59:36,170 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 04:59:37,060 [236] [DEBUG] [app] Starting request: urn:request:0410faa5-cd0d-4a96-bfbb-cdcb0939520e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:37,060 [231] [DEBUG] [app] Starting request: urn:request:2dd7f28e-f975-4e76-94aa-0c1be5227569 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:37,065 [242] [DEBUG] [app] Starting request: urn:request:61371252-7b36-4e00-b32b-f437e35be198 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:37,065 [230] [DEBUG] [app] Starting request: urn:request:12bbfa1e-21b3-4468-87db-30a38a2a985b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:37,065 [242] [DEBUG] [app] Ending request: urn:request:61371252-7b36-4e00-b32b-f437e35be198 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:61371252-7b36-4e00-b32b-f437e35be198', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 04:59:37,065 [230] [DEBUG] [app] Ending request: urn:request:12bbfa1e-21b3-4468-87db-30a38a2a985b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:12bbfa1e-21b3-4468-87db-30a38a2a985b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 04:59:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:59:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 04:59:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:37,070 [231] [DEBUG] [app] Starting request: urn:request:88f2a3c6-023b-43e1-9a3f-6165daf43f1c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:37,070 [231] [DEBUG] [app] Ending request: urn:request:88f2a3c6-023b-43e1-9a3f-6165daf43f1c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:88f2a3c6-023b-43e1-9a3f-6165daf43f1c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:37,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:37,072 [235] [DEBUG] [app] Starting request: urn:request:8c55e8d3-6e91-4a6a-8aac-5dd16a77ad57 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:37,072 [235] [DEBUG] [app] Ending request: urn:request:8c55e8d3-6e91-4a6a-8aac-5dd16a77ad57 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c55e8d3-6e91-4a6a-8aac-5dd16a77ad57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:37,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:37,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:37,089 [231] [DEBUG] [app] Ending request: urn:request:2dd7f28e-f975-4e76-94aa-0c1be5227569 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2dd7f28e-f975-4e76-94aa-0c1be5227569', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:37,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) gunicorn-web stdout | 2023-01-17 04:59:37,089 [236] [DEBUG] [app] Ending request: urn:request:0410faa5-cd0d-4a96-bfbb-cdcb0939520e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0410faa5-cd0d-4a96-bfbb-cdcb0939520e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:37,089 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 04:59:37,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 04:59:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:59:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:37.537407+00:00 (in 0.280337 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:59:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:37 UTC)" (scheduled at 2023-01-17 04:59:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:59:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 04:59:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 37, 257497), True, datetime.datetime(2023, 1, 17, 4, 59, 37, 257497), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 04:59:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 04:59:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 04:59:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 04:59:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 04:59:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:37.256652+00:00 (in 59.718830 seconds) securityscanningnotificationworker stdout | 2023-01-17 04:59:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:37 UTC)" (scheduled at 2023-01-17 04:59:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 04:59:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 04:59:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 04:59:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 04:59:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:10.738568+00:00 (in 29.999519 seconds) buildlogsarchiver stdout | 2023-01-17 04:59:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:10 UTC)" (scheduled at 2023-01-17 04:59:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 04:59:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 4, 59, 40, 739420), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 04:59:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 04:59:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 04:59:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:59:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:59:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:42.941536+00:00 (in 1.792824 seconds) namespacegcworker stdout | 2023-01-17 04:59:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:41 UTC)" (scheduled at 2023-01-17 04:59:41.148245+00:00) namespacegcworker stdout | 2023-01-17 04:59:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 04:59:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 41, 149127), True, datetime.datetime(2023, 1, 17, 4, 59, 41, 149127), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 04:59:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 04:59:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 04:59:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 04:59:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 04:59:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:41.148245+00:00 (in 58.206228 seconds) namespacegcworker stdout | 2023-01-17 04:59:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:42 UTC)" (scheduled at 2023-01-17 04:59:42.941536+00:00) namespacegcworker stdout | 2023-01-17 04:59:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 04:59:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:00:42 UTC)" executed successfully builder stdout | 2023-01-17 04:59:45,239 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:45,251 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:45,252 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:45,262 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:55.646354+00:00 (in 9.999525 seconds) notificationworker stdout | 2023-01-17 04:59:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:55 UTC)" (scheduled at 2023-01-17 04:59:45.646354+00:00) notificationworker stdout | 2023-01-17 04:59:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 45, 647262), True, datetime.datetime(2023, 1, 17, 4, 59, 45, 647262), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 04:59:55 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 04:59:49.537472+00:00 (in 2.600032 seconds) securityworker stdout | 2023-01-17 04:59:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:16 UTC)" (scheduled at 2023-01-17 04:59:46.936988+00:00) securityworker stdout | 2023-01-17 04:59:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:16 UTC)" executed successfully securityworker stdout | 2023-01-17 04:59:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 04:59:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:16.936988+00:00 (in 27.399053 seconds) securityworker stdout | 2023-01-17 04:59:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:19 UTC)" (scheduled at 2023-01-17 04:59:49.537472+00:00) securityworker stdout | 2023-01-17 04:59:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 04:59:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:49,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 04:59:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 04:59:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 04:59:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 04:59:49,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 04:59:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 04:59:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 04:59:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 04:59:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 04:59:49,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 04:59:49,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 04:59:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 04:59:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:19 UTC)" executed successfully gcworker stdout | 2023-01-17 04:59:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 04:59:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:21.348605+00:00 (in 29.999549 seconds) gcworker stdout | 2023-01-17 04:59:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:21 UTC)" (scheduled at 2023-01-17 04:59:51.348605+00:00) gcworker stdout | 2023-01-17 04:59:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672721991349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 04:59:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 04:59:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 04:59:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 04:59:52,060 [231] [DEBUG] [app] Starting request: urn:request:57a04670-1e98-4f3d-bc8c-34f9e169420b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:52,061 [236] [DEBUG] [app] Starting request: urn:request:9e093669-ebab-47ff-b750-017bc906124f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 04:59:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 04:59:52,065 [243] [DEBUG] [app] Starting request: urn:request:c9c47a02-5b00-4b4a-b4b2-62447cc1d7a1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:52,066 [230] [DEBUG] [app] Starting request: urn:request:398609d4-a820-45c2-8f28-e6c69b0aaf16 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 04:59:52,066 [243] [DEBUG] [app] Ending request: urn:request:c9c47a02-5b00-4b4a-b4b2-62447cc1d7a1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c9c47a02-5b00-4b4a-b4b2-62447cc1d7a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:52,066 [230] [DEBUG] [app] Ending request: urn:request:398609d4-a820-45c2-8f28-e6c69b0aaf16 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:398609d4-a820-45c2-8f28-e6c69b0aaf16', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 04:59:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 04:59:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 04:59:52,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 04:59:52,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,069 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 04:59:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:52,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 04:59:52,071 [233] [DEBUG] [app] Starting request: urn:request:996863d3-5134-41e6-abbb-20f2898ef8a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:52,072 [235] [DEBUG] [app] Starting request: urn:request:fa504938-0508-4a9d-8cec-9bd304b1eb81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 04:59:52,072 [233] [DEBUG] [app] Ending request: urn:request:996863d3-5134-41e6-abbb-20f2898ef8a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:996863d3-5134-41e6-abbb-20f2898ef8a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 04:59:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 04:59:52,072 [235] [DEBUG] [app] Ending request: urn:request:fa504938-0508-4a9d-8cec-9bd304b1eb81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa504938-0508-4a9d-8cec-9bd304b1eb81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 04:59:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:52,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 04:59:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:52,073 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 04:59:52,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 04:59:52,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:52,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 04:59:52,089 [231] [DEBUG] [app] Ending request: urn:request:57a04670-1e98-4f3d-bc8c-34f9e169420b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:57a04670-1e98-4f3d-bc8c-34f9e169420b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:52,089 [236] [DEBUG] [app] Ending request: urn:request:9e093669-ebab-47ff-b750-017bc906124f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e093669-ebab-47ff-b750-017bc906124f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 04:59:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 04:59:52,089 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-web stdout | 2023-01-17 04:59:52,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 04:59:52,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:04:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 04:59:55,265 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 04:59:55,278 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 04:59:55,278 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 04:59:55,289 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 04:59:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:01.636362+00:00 (in 5.989526 seconds) notificationworker stdout | 2023-01-17 04:59:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:05 UTC)" (scheduled at 2023-01-17 04:59:55.646354+00:00) notificationworker stdout | 2023-01-17 04:59:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 04:59:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 4, 59, 55, 647287), True, datetime.datetime(2023, 1, 17, 4, 59, 55, 647287), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 04:59:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 04:59:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 04:59:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 04:59:57,207 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 04:59:57,218 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 04:59:58,120 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 04:59:58,621 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 04:59:58,706 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 04:59:59,023 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 04:59:59,145 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:00:00,596 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:00:00,598 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:00:00,601 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:00:00,603 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:00:00,605 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:00:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:05.646354+00:00 (in 4.009514 seconds) notificationworker stdout | 2023-01-17 05:00:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:01 UTC)" (scheduled at 2023-01-17 05:00:01.636362+00:00) notificationworker stdout | 2023-01-17 05:00:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:00:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:00:02,169 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:00:02,557 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:00:03,684 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:00:03,717 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:00:03,926 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:00:04,458 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:00:04,603 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:00:04,607 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:00:04,610 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:00:04,643 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:00:04,824 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:00:04,829 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:00:04,832 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:00:04,836 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:00:04,838 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:00:04,840 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:00:04,844 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:00:04,846 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:00:04,848 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} securityscanningnotificationworker stdout | 2023-01-17 05:00:04,925 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositorygcworker stdout | 2023-01-17 05:00:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:00:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:19.346120+00:00 (in 14.408830 seconds) repositorygcworker stdout | 2023-01-17 05:00:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:04 UTC)" (scheduled at 2023-01-17 05:00:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:00:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:00:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 4, 937722), True, datetime.datetime(2023, 1, 17, 5, 0, 4, 937722), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:00:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:00:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:00:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:04 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:00:05,095 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 05:00:05,299 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:05,311 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:05,312 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:05,322 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 05:00:05,415 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:00:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:15.646354+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 05:00:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:15 UTC)" (scheduled at 2023-01-17 05:00:05.646354+00:00) notificationworker stdout | 2023-01-17 05:00:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 5, 647332), True, datetime.datetime(2023, 1, 17, 5, 0, 5, 647332), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:15 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:00:05,928 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:00:06,093 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:00:06,205 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:00:07,060 [231] [DEBUG] [app] Starting request: urn:request:1556f37c-0242-450e-91e2-9c88a2a511d8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:07,060 [233] [DEBUG] [app] Starting request: urn:request:8866513e-ddd9-4a31-a4ac-656a7030aad2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:07,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:07,066 [243] [DEBUG] [app] Starting request: urn:request:5a20a4fb-cf68-4202-8b59-f8a5f3918e2f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:07,066 [242] [DEBUG] [app] Starting request: urn:request:f1fe15f2-5754-4786-a364-929cab037758 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:07,066 [243] [DEBUG] [app] Ending request: urn:request:5a20a4fb-cf68-4202-8b59-f8a5f3918e2f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a20a4fb-cf68-4202-8b59-f8a5f3918e2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:00:07,066 [242] [DEBUG] [app] Ending request: urn:request:f1fe15f2-5754-4786-a364-929cab037758 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f1fe15f2-5754-4786-a364-929cab037758', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:00:07,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:07,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:00:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:07,071 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:07,071 [235] [DEBUG] [app] Starting request: urn:request:6b035c27-c6e8-4376-b887-3ddd4732a701 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:07,071 [235] [DEBUG] [app] Ending request: urn:request:6b035c27-c6e8-4376-b887-3ddd4732a701 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b035c27-c6e8-4376-b887-3ddd4732a701', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:07,072 [231] [DEBUG] [app] Starting request: urn:request:2dad0eaf-f282-4abd-8860-e66b2b211b53 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:00:07,072 [231] [DEBUG] [app] Ending request: urn:request:2dad0eaf-f282-4abd-8860-e66b2b211b53 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2dad0eaf-f282-4abd-8860-e66b2b211b53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:00:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:07,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:07,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:07,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:07,084 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:07,089 [233] [DEBUG] [app] Ending request: urn:request:8866513e-ddd9-4a31-a4ac-656a7030aad2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8866513e-ddd9-4a31-a4ac-656a7030aad2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:07,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:07,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:00:07,093 [231] [DEBUG] [app] Ending request: urn:request:1556f37c-0242-450e-91e2-9c88a2a511d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1556f37c-0242-450e-91e2-9c88a2a511d8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:07,093 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.034 47 0.034) gunicorn-web stdout | 2023-01-17 05:00:07,093 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 05:00:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:00:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:40.738568+00:00 (in 29.999518 seconds) buildlogsarchiver stdout | 2023-01-17 05:00:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:40 UTC)" (scheduled at 2023-01-17 05:00:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:00:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 0, 10, 739467), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:00:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:00:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:00:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:40 UTC)" executed successfully builder stdout | 2023-01-17 05:00:15,333 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:15,345 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:15,345 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:15,356 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:25.646354+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 05:00:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:25 UTC)" (scheduled at 2023-01-17 05:00:15.646354+00:00) notificationworker stdout | 2023-01-17 05:00:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 15, 647287), True, datetime.datetime(2023, 1, 17, 5, 0, 15, 647287), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:19.537472+00:00 (in 2.600018 seconds) securityworker stdout | 2023-01-17 05:00:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:46 UTC)" (scheduled at 2023-01-17 05:00:16.936988+00:00) securityworker stdout | 2023-01-17 05:00:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:00:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:00:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:04.936835+00:00 (in 45.590236 seconds) repositorygcworker stdout | 2023-01-17 05:00:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:19 UTC)" (scheduled at 2023-01-17 05:00:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:00:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:00:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:46.936988+00:00 (in 27.399027 seconds) securityworker stdout | 2023-01-17 05:00:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:49 UTC)" (scheduled at 2023-01-17 05:00:19.537472+00:00) securityworker stdout | 2023-01-17 05:00:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:00:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:00:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:00:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:00:19,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:00:19,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:00:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:00:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:00:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:51.348605+00:00 (in 29.999525 seconds) gcworker stdout | 2023-01-17 05:00:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:51 UTC)" (scheduled at 2023-01-17 05:00:21.348605+00:00) gcworker stdout | 2023-01-17 05:00:21,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:00:21,360 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722021359, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:00:21,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:00:21,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:00:21,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:00:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:00:22,060 [235] [DEBUG] [app] Starting request: urn:request:446013c9-1499-4c2e-9bee-5d1cd98e84e0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:22,060 [231] [DEBUG] [app] Starting request: urn:request:5046b2da-d428-45e4-965c-5c6bd29b2986 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:22,066 [242] [DEBUG] [app] Starting request: urn:request:ea013542-367d-449f-b861-192d0ee68d4b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:22,066 [243] [DEBUG] [app] Starting request: urn:request:8a529e6a-6c18-4b72-83dc-94fc6ed0d636 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:22,066 [242] [DEBUG] [app] Ending request: urn:request:ea013542-367d-449f-b861-192d0ee68d4b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea013542-367d-449f-b861-192d0ee68d4b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:00:22,066 [243] [DEBUG] [app] Ending request: urn:request:8a529e6a-6c18-4b72-83dc-94fc6ed0d636 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a529e6a-6c18-4b72-83dc-94fc6ed0d636', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:00:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:22,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:22,071 [233] [DEBUG] [app] Starting request: urn:request:a3244852-0a8d-4b4e-91c5-82553991d0b7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:22,071 [233] [DEBUG] [app] Ending request: urn:request:a3244852-0a8d-4b4e-91c5-82553991d0b7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a3244852-0a8d-4b4e-91c5-82553991d0b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:22,071 [231] [DEBUG] [app] Starting request: urn:request:0746c87c-012c-4c58-b1d3-7c763b4c11b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:00:22,072 [231] [DEBUG] [app] Ending request: urn:request:0746c87c-012c-4c58-b1d3-7c763b4c11b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0746c87c-012c-4c58-b1d3-7c763b4c11b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:22,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:00:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:22,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:22,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:22,089 [235] [DEBUG] [app] Ending request: urn:request:446013c9-1499-4c2e-9bee-5d1cd98e84e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:446013c9-1499-4c2e-9bee-5d1cd98e84e0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:22,089 [231] [DEBUG] [app] Ending request: urn:request:5046b2da-d428-45e4-965c-5c6bd29b2986 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5046b2da-d428-45e4-965c-5c6bd29b2986', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:00:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:00:25,358 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:25,370 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:25,371 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:25,381 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:35.646354+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 05:00:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:35 UTC)" (scheduled at 2023-01-17 05:00:25.646354+00:00) notificationworker stdout | 2023-01-17 05:00:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 25, 647231), True, datetime.datetime(2023, 1, 17, 5, 0, 25, 647231), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:00:27,243 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:00:27,248 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:00:28,129 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:00:28,657 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:00:28,732 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} exportactionlogsworker stdout | 2023-01-17 05:00:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:00:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:35.853396+00:00 (in 6.889137 seconds) exportactionlogsworker stdout | 2023-01-17 05:00:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:28 UTC)" (scheduled at 2023-01-17 05:00:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:00:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:00:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:28 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:00:29,041 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:00:29,159 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:00:30,603 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:00:30,607 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:00:30,609 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:00:30,612 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:00:30,614 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:00:32,204 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:00:32,590 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:00:33,720 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:00:33,752 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:00:33,932 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:00:34,464 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:00:34,610 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:00:34,614 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:00:34,616 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:00:34,667 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:00:34,831 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:00:34,837 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:00:34,839 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:00:34,842 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:00:34,845 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:00:34,847 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:00:34,851 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:00:34,854 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:00:34,857 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} securityscanningnotificationworker stdout | 2023-01-17 05:00:34,959 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:00:35,123 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 05:00:35,388 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:35,404 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:35,404 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:35,415 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 05:00:35,450 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:00:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:45.646354+00:00 (in 9.999602 seconds) notificationworker stdout | 2023-01-17 05:00:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:45 UTC)" (scheduled at 2023-01-17 05:00:35.646354+00:00) notificationworker stdout | 2023-01-17 05:00:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 35, 647181), True, datetime.datetime(2023, 1, 17, 5, 0, 35, 647181), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:00:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:00:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:28.963662+00:00 (in 53.109774 seconds) exportactionlogsworker stdout | 2023-01-17 05:00:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:35 UTC)" (scheduled at 2023-01-17 05:00:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:00:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:00:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 35, 854353), True, datetime.datetime(2023, 1, 17, 5, 0, 35, 854353), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:00:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:00:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:00:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:00:35,964 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:00:36,122 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:00:36,238 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:00:37,060 [231] [DEBUG] [app] Starting request: urn:request:bc148dfe-2398-4ae8-ae56-7c48511eab6f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:37,060 [235] [DEBUG] [app] Starting request: urn:request:4a7d878a-7296-48f3-97a8-e14154df2747 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:37,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:37,065 [230] [DEBUG] [app] Starting request: urn:request:cde7e209-72e3-4777-8374-69fb016b473e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:37,066 [243] [DEBUG] [app] Starting request: urn:request:123cf538-ec6a-45e8-bdb3-a1cfc66f2fd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:37,066 [230] [DEBUG] [app] Ending request: urn:request:cde7e209-72e3-4777-8374-69fb016b473e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cde7e209-72e3-4777-8374-69fb016b473e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:00:37,066 [243] [DEBUG] [app] Ending request: urn:request:123cf538-ec6a-45e8-bdb3-a1cfc66f2fd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:123cf538-ec6a-45e8-bdb3-a1cfc66f2fd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:00:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:00:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:37,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:37,070 [233] [DEBUG] [app] Starting request: urn:request:40fc93c8-a0bb-4ef7-874e-4f23bcf14574 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:37,070 [236] [DEBUG] [app] Starting request: urn:request:31611337-1fcc-4a5b-b0ea-7f70fa2e3283 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:37,071 [233] [DEBUG] [app] Ending request: urn:request:40fc93c8-a0bb-4ef7-874e-4f23bcf14574 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:40fc93c8-a0bb-4ef7-874e-4f23bcf14574', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:37,071 [236] [DEBUG] [app] Ending request: urn:request:31611337-1fcc-4a5b-b0ea-7f70fa2e3283 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31611337-1fcc-4a5b-b0ea-7f70fa2e3283', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:37,087 [231] [DEBUG] [app] Ending request: urn:request:bc148dfe-2398-4ae8-ae56-7c48511eab6f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bc148dfe-2398-4ae8-ae56-7c48511eab6f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:37,087 [235] [DEBUG] [app] Ending request: urn:request:4a7d878a-7296-48f3-97a8-e14154df2747 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4a7d878a-7296-48f3-97a8-e14154df2747', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:37,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:00:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:00:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:00:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:37.537407+00:00 (in 0.280340 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:00:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:37 UTC)" (scheduled at 2023-01-17 05:00:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:00:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:00:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 37, 257481), True, datetime.datetime(2023, 1, 17, 5, 0, 37, 257481), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:00:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:00:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:00:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:00:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:00:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:23.537048+00:00 (in 45.999272 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:00:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:37 UTC)" (scheduled at 2023-01-17 05:00:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:00:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:00:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:00:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:00:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:10.738568+00:00 (in 29.999514 seconds) buildlogsarchiver stdout | 2023-01-17 05:00:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:10 UTC)" (scheduled at 2023-01-17 05:00:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:00:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 0, 40, 739456), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:00:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:00:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:00:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:00:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:00:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:42.941536+00:00 (in 1.792829 seconds) namespacegcworker stdout | 2023-01-17 05:00:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:41 UTC)" (scheduled at 2023-01-17 05:00:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:00:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:00:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 41, 149121), True, datetime.datetime(2023, 1, 17, 5, 0, 41, 149121), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:00:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:00:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:00:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:00:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:00:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:41.148245+00:00 (in 58.206251 seconds) namespacegcworker stdout | 2023-01-17 05:00:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:42 UTC)" (scheduled at 2023-01-17 05:00:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:00:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:00:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:01:42 UTC)" executed successfully builder stdout | 2023-01-17 05:00:45,425 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:45,438 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:45,438 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:45,449 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:55.646354+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 05:00:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:55 UTC)" (scheduled at 2023-01-17 05:00:45.646354+00:00) notificationworker stdout | 2023-01-17 05:00:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 45, 647208), True, datetime.datetime(2023, 1, 17, 5, 0, 45, 647208), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:00:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:00:49.537472+00:00 (in 2.600031 seconds) securityworker stdout | 2023-01-17 05:00:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:16 UTC)" (scheduled at 2023-01-17 05:00:46.936988+00:00) securityworker stdout | 2023-01-17 05:00:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:00:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:00:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:16.936988+00:00 (in 27.399050 seconds) securityworker stdout | 2023-01-17 05:00:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:19 UTC)" (scheduled at 2023-01-17 05:00:49.537472+00:00) securityworker stdout | 2023-01-17 05:00:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:00:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:00:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:00:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:00:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:00:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:00:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:00:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:00:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:00:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:00:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:00:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:00:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:00:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:00:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:00:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:21.348605+00:00 (in 29.999507 seconds) gcworker stdout | 2023-01-17 05:00:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:21 UTC)" (scheduled at 2023-01-17 05:00:51.348605+00:00) gcworker stdout | 2023-01-17 05:00:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722051349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:00:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:00:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:00:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:00:52,060 [235] [DEBUG] [app] Starting request: urn:request:f03db089-810e-42fa-b889-0af9025d4e83 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:52,060 [231] [DEBUG] [app] Starting request: urn:request:bb9924a2-cd1d-47c7-a095-bfacb8c4e5e4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:00:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:00:52,065 [230] [DEBUG] [app] Starting request: urn:request:1a9b9c3b-418a-4eab-893f-6d1407980e78 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:52,065 [243] [DEBUG] [app] Starting request: urn:request:b3d004c0-273a-4998-94ac-97c478e1a262 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:00:52,066 [230] [DEBUG] [app] Ending request: urn:request:1a9b9c3b-418a-4eab-893f-6d1407980e78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1a9b9c3b-418a-4eab-893f-6d1407980e78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:00:52,066 [243] [DEBUG] [app] Ending request: urn:request:b3d004c0-273a-4998-94ac-97c478e1a262 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b3d004c0-273a-4998-94ac-97c478e1a262', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:00:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:00:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:00:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:00:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:00:52,070 [236] [DEBUG] [app] Starting request: urn:request:bcae7640-02ef-4853-8a3d-7acd439dc9c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:52,070 [233] [DEBUG] [app] Starting request: urn:request:8a64cd34-de63-47c7-815d-038255c2e782 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:00:52,071 [236] [DEBUG] [app] Ending request: urn:request:bcae7640-02ef-4853-8a3d-7acd439dc9c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bcae7640-02ef-4853-8a3d-7acd439dc9c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:00:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:52,071 [233] [DEBUG] [app] Ending request: urn:request:8a64cd34-de63-47c7-815d-038255c2e782 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8a64cd34-de63-47c7-815d-038255c2e782', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:00:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:00:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:00:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:00:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:00:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:00:52,087 [235] [DEBUG] [app] Ending request: urn:request:f03db089-810e-42fa-b889-0af9025d4e83 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f03db089-810e-42fa-b889-0af9025d4e83', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:52,087 [231] [DEBUG] [app] Ending request: urn:request:bb9924a2-cd1d-47c7-a095-bfacb8c4e5e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb9924a2-cd1d-47c7-a095-bfacb8c4e5e4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:00:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:00:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:00:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:00:55,450 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:00:55,462 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:00:55,462 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:00:55,473 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:00:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:01.636362+00:00 (in 5.989581 seconds) notificationworker stdout | 2023-01-17 05:00:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:05 UTC)" (scheduled at 2023-01-17 05:00:55.646354+00:00) notificationworker stdout | 2023-01-17 05:00:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:00:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 0, 55, 647236), True, datetime.datetime(2023, 1, 17, 5, 0, 55, 647236), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:00:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:00:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:00:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:00:57,249 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:00:57,265 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:00:58,150 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:00:58,693 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:00:58,741 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:00:59,077 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:00:59,177 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:01:00,611 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:01:00,613 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:01:00,615 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:01:00,617 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:01:00,620 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:01:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:05.646354+00:00 (in 4.009530 seconds) notificationworker stdout | 2023-01-17 05:01:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:01 UTC)" (scheduled at 2023-01-17 05:01:01.636362+00:00) notificationworker stdout | 2023-01-17 05:01:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:01:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:01:02,240 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:01:02,623 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:01:03,730 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:01:03,788 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:01:03,967 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:01:04,472 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:01:04,617 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:01:04,620 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:01:04,622 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:01:04,698 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:01:04,842 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:01:04,844 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:01:04,847 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:01:04,850 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:01:04,852 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:01:04,855 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:01:04,857 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:01:04,865 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:01:04,867 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} repositorygcworker stdout | 2023-01-17 05:01:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:01:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:17.345601+00:00 (in 12.408314 seconds) repositorygcworker stdout | 2023-01-17 05:01:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:04 UTC)" (scheduled at 2023-01-17 05:01:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:01:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:01:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 4, 937746), True, datetime.datetime(2023, 1, 17, 5, 1, 4, 937746), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:01:04,961 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:01:04,961 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:01:04,961 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:04 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:01:04,969 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:01:05,149 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 05:01:05,482 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue globalpromstats stdout | 2023-01-17 05:01:05,486 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 05:01:05,494 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:05,495 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:05,505 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:15.646354+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:01:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:15 UTC)" (scheduled at 2023-01-17 05:01:05.646354+00:00) notificationworker stdout | 2023-01-17 05:01:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 5, 647249), True, datetime.datetime(2023, 1, 17, 5, 1, 5, 647249), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:15 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:01:06,000 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:01:06,157 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:01:06,273 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:01:07,060 [236] [DEBUG] [app] Starting request: urn:request:29d63c5e-3ab4-4aa0-8a24-444c8e45c8fa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:07,060 [231] [DEBUG] [app] Starting request: urn:request:722d3263-0563-4db1-a950-e731b3d13c86 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:07,065 [243] [DEBUG] [app] Starting request: urn:request:30e81223-8cfe-46a3-9c40-55987eb8e045 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:07,066 [230] [DEBUG] [app] Starting request: urn:request:47949512-124b-4b6e-bc5f-7ab7b612df07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:07,066 [243] [DEBUG] [app] Ending request: urn:request:30e81223-8cfe-46a3-9c40-55987eb8e045 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30e81223-8cfe-46a3-9c40-55987eb8e045', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:01:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:01:07,066 [230] [DEBUG] [app] Ending request: urn:request:47949512-124b-4b6e-bc5f-7ab7b612df07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:47949512-124b-4b6e-bc5f-7ab7b612df07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:01:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:01:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:07,071 [235] [DEBUG] [app] Starting request: urn:request:7f0cdae2-aeda-44ee-99db-25d450e8a0d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:07,071 [236] [DEBUG] [app] Starting request: urn:request:a294e301-11ba-42df-8371-23c4b6ab1292 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:07,071 [235] [DEBUG] [app] Ending request: urn:request:7f0cdae2-aeda-44ee-99db-25d450e8a0d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f0cdae2-aeda-44ee-99db-25d450e8a0d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:07,071 [236] [DEBUG] [app] Ending request: urn:request:a294e301-11ba-42df-8371-23c4b6ab1292 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a294e301-11ba-42df-8371-23c4b6ab1292', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:07,088 [236] [DEBUG] [app] Ending request: urn:request:29d63c5e-3ab4-4aa0-8a24-444c8e45c8fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29d63c5e-3ab4-4aa0-8a24-444c8e45c8fa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:01:07,092 [231] [DEBUG] [app] Ending request: urn:request:722d3263-0563-4db1-a950-e731b3d13c86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:722d3263-0563-4db1-a950-e731b3d13c86', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:07,093 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:07,093 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.033 47 0.033) buildlogsarchiver stdout | 2023-01-17 05:01:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:01:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:40.738568+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 05:01:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:40 UTC)" (scheduled at 2023-01-17 05:01:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:01:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 1, 10, 739403), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:01:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:01:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:01:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:40 UTC)" executed successfully builder stdout | 2023-01-17 05:01:15,515 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:15,528 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:15,528 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:15,538 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:25.646354+00:00 (in 9.999572 seconds) notificationworker stdout | 2023-01-17 05:01:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:25 UTC)" (scheduled at 2023-01-17 05:01:15.646354+00:00) notificationworker stdout | 2023-01-17 05:01:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 15, 647186), True, datetime.datetime(2023, 1, 17, 5, 1, 15, 647186), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:19.537472+00:00 (in 2.600025 seconds) securityworker stdout | 2023-01-17 05:01:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:46 UTC)" (scheduled at 2023-01-17 05:01:16.936988+00:00) securityworker stdout | 2023-01-17 05:01:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:01:17,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:01:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:19.346120+00:00 (in 1.999910 seconds) repositorygcworker stdout | 2023-01-17 05:01:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:17 UTC)" (scheduled at 2023-01-17 05:01:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:01:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 1, 17, 346627), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:01:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 17, 346627), True, datetime.datetime(2023, 1, 17, 5, 1, 17, 346627), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:01:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 17, 346627), True, datetime.datetime(2023, 1, 17, 5, 1, 17, 346627), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 1, 17, 346627), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:01:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:01:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:01:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:01:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:04.936835+00:00 (in 45.590232 seconds) repositorygcworker stdout | 2023-01-17 05:01:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:19 UTC)" (scheduled at 2023-01-17 05:01:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:01:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:01:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:46.936988+00:00 (in 27.399089 seconds) securityworker stdout | 2023-01-17 05:01:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:49 UTC)" (scheduled at 2023-01-17 05:01:19.537472+00:00) securityworker stdout | 2023-01-17 05:01:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:01:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:01:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:01:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:01:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:01:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:01:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:01:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:01:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:51.348605+00:00 (in 29.999527 seconds) gcworker stdout | 2023-01-17 05:01:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:51 UTC)" (scheduled at 2023-01-17 05:01:21.348605+00:00) gcworker stdout | 2023-01-17 05:01:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722081349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:01:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:01:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:01:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:01:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:01:22,060 [231] [DEBUG] [app] Starting request: urn:request:95ba0e04-81e7-4baa-8525-e029285c0d6f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:22,060 [235] [DEBUG] [app] Starting request: urn:request:9f9296f4-2323-43e1-83ee-7e9924e3d8db (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:22,065 [243] [DEBUG] [app] Starting request: urn:request:40d8c371-44fa-48f2-bb92-59fc34d0bb94 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:22,065 [230] [DEBUG] [app] Starting request: urn:request:d94301d5-e3fd-4046-b6fb-4f9dd2e5876d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:22,065 [243] [DEBUG] [app] Ending request: urn:request:40d8c371-44fa-48f2-bb92-59fc34d0bb94 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40d8c371-44fa-48f2-bb92-59fc34d0bb94', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:01:22,065 [230] [DEBUG] [app] Ending request: urn:request:d94301d5-e3fd-4046-b6fb-4f9dd2e5876d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d94301d5-e3fd-4046-b6fb-4f9dd2e5876d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:01:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:22,069 [235] [DEBUG] [app] Starting request: urn:request:46e942ab-d5ac-4b5e-8118-98cf63b173c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:22,070 [235] [DEBUG] [app] Ending request: urn:request:46e942ab-d5ac-4b5e-8118-98cf63b173c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:46e942ab-d5ac-4b5e-8118-98cf63b173c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:22,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:01:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:22,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:22,071 [233] [DEBUG] [app] Starting request: urn:request:99acb7ad-8f53-4aad-9fbb-acce7539b507 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:22,072 [233] [DEBUG] [app] Ending request: urn:request:99acb7ad-8f53-4aad-9fbb-acce7539b507 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:99acb7ad-8f53-4aad-9fbb-acce7539b507', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:01:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:22,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:22,086 [231] [DEBUG] [app] Ending request: urn:request:95ba0e04-81e7-4baa-8525-e029285c0d6f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:95ba0e04-81e7-4baa-8525-e029285c0d6f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:22,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:01:22,088 [235] [DEBUG] [app] Ending request: urn:request:9f9296f4-2323-43e1-83ee-7e9924e3d8db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f9296f4-2323-43e1-83ee-7e9924e3d8db', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:01:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:01:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:37.256652+00:00 (in 13.719200 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:01:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:23 UTC)" (scheduled at 2023-01-17 05:01:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:01:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 1, 23, 537837), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:01:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 23, 537837), True, datetime.datetime(2023, 1, 17, 5, 1, 23, 537837), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:01:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 23, 537837), True, datetime.datetime(2023, 1, 17, 5, 1, 23, 537837), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 1, 23, 537837), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:01:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:01:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:23 UTC)" executed successfully builder stdout | 2023-01-17 05:01:25,549 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:25,561 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:25,561 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:25,572 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:35.646354+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 05:01:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:35 UTC)" (scheduled at 2023-01-17 05:01:25.646354+00:00) notificationworker stdout | 2023-01-17 05:01:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 25, 647195), True, datetime.datetime(2023, 1, 17, 5, 1, 25, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:35 UTC)" executed successfully gcworker stdout | 2023-01-17 05:01:27,282 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:01:27,285 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:01:28,181 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:01:28,727 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:01:28,749 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} exportactionlogsworker stdout | 2023-01-17 05:01:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:01:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:35.853396+00:00 (in 6.889252 seconds) exportactionlogsworker stdout | 2023-01-17 05:01:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:28 UTC)" (scheduled at 2023-01-17 05:01:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:01:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:01:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:28 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:01:29,113 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:01:29,183 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:01:30,618 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:01:30,622 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:01:30,624 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:01:30,627 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:01:30,629 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:01:32,271 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:01:32,636 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:01:33,766 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:01:33,824 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:01:33,998 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:01:34,496 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:01:34,624 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:01:34,626 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:01:34,628 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:01:34,729 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:01:34,851 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:01:34,855 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:01:34,857 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:01:34,860 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:01:34,862 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:01:34,865 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:01:34,867 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:01:34,874 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:01:34,876 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} securityscanningnotificationworker stdout | 2023-01-17 05:01:35,004 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:01:35,162 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:01:35,519 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 05:01:35,582 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:35,594 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:35,595 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:35,605 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:45.646354+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 05:01:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:45 UTC)" (scheduled at 2023-01-17 05:01:35.646354+00:00) notificationworker stdout | 2023-01-17 05:01:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 35, 647231), True, datetime.datetime(2023, 1, 17, 5, 1, 35, 647231), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:01:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:01:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:54.963413+00:00 (in 19.109530 seconds) exportactionlogsworker stdout | 2023-01-17 05:01:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:35 UTC)" (scheduled at 2023-01-17 05:01:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:01:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:01:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 35, 854288), True, datetime.datetime(2023, 1, 17, 5, 1, 35, 854288), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:01:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:01:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:01:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:01:36,005 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:01:36,192 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:01:36,308 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:01:37,059 [235] [DEBUG] [app] Starting request: urn:request:b94846d0-fee5-4a6b-bd86-4adbadd7d1d2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:37,059 [231] [DEBUG] [app] Starting request: urn:request:ac14b6ee-8acb-4c2a-882d-d69561b0740f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:37,064 [242] [DEBUG] [app] Starting request: urn:request:28cef4bf-ded2-4ad6-aaf7-5bba5e1c4822 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:37,064 [230] [DEBUG] [app] Starting request: urn:request:cf0f34f5-0a19-4117-bfb2-9e1a84436f32 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:37,064 [242] [DEBUG] [app] Ending request: urn:request:28cef4bf-ded2-4ad6-aaf7-5bba5e1c4822 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28cef4bf-ded2-4ad6-aaf7-5bba5e1c4822', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:01:37,065 [230] [DEBUG] [app] Ending request: urn:request:cf0f34f5-0a19-4117-bfb2-9e1a84436f32 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf0f34f5-0a19-4117-bfb2-9e1a84436f32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:01:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:01:37,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:37,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:37,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:37,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:37,069 [231] [DEBUG] [app] Starting request: urn:request:07de0161-ef66-406e-81c4-76c7b682fc67 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:37,069 [233] [DEBUG] [app] Starting request: urn:request:821fa6c4-e4cf-45cc-8d30-a4ea454f92d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:37,069 [231] [DEBUG] [app] Ending request: urn:request:07de0161-ef66-406e-81c4-76c7b682fc67 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:07de0161-ef66-406e-81c4-76c7b682fc67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:37,070 [233] [DEBUG] [app] Ending request: urn:request:821fa6c4-e4cf-45cc-8d30-a4ea454f92d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:821fa6c4-e4cf-45cc-8d30-a4ea454f92d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:37,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:37,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:01:37,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:37,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:37,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:37,086 [231] [DEBUG] [app] Ending request: urn:request:ac14b6ee-8acb-4c2a-882d-d69561b0740f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac14b6ee-8acb-4c2a-882d-d69561b0740f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:37,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:37,086 [235] [DEBUG] [app] Ending request: urn:request:b94846d0-fee5-4a6b-bd86-4adbadd7d1d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b94846d0-fee5-4a6b-bd86-4adbadd7d1d2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:37,086 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:37,086 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:01:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:01:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:01:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:37.537407+00:00 (in 0.280279 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:01:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:37 UTC)" (scheduled at 2023-01-17 05:01:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:01:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:01:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 37, 257573), True, datetime.datetime(2023, 1, 17, 5, 1, 37, 257573), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:01:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:01:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:01:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:01:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:01:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:37.256652+00:00 (in 59.718809 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:01:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:37 UTC)" (scheduled at 2023-01-17 05:01:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:01:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:01:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:01:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:01:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:10.738568+00:00 (in 29.999555 seconds) buildlogsarchiver stdout | 2023-01-17 05:01:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:10 UTC)" (scheduled at 2023-01-17 05:01:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:01:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 1, 40, 739404), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:01:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:01:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:01:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:01:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:01:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:42.941536+00:00 (in 1.792820 seconds) namespacegcworker stdout | 2023-01-17 05:01:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:41 UTC)" (scheduled at 2023-01-17 05:01:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:01:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:01:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 41, 149144), True, datetime.datetime(2023, 1, 17, 5, 1, 41, 149144), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:01:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:01:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:01:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:01:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:01:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:06.941185+00:00 (in 23.999162 seconds) namespacegcworker stdout | 2023-01-17 05:01:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:42 UTC)" (scheduled at 2023-01-17 05:01:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:01:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:01:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:02:42 UTC)" executed successfully builder stdout | 2023-01-17 05:01:45,615 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:01:45,628 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:45,628 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:01:45,639 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:55.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:01:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:55 UTC)" (scheduled at 2023-01-17 05:01:45.646354+00:00) notificationworker stdout | 2023-01-17 05:01:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 45, 647205), True, datetime.datetime(2023, 1, 17, 5, 1, 45, 647205), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:01:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:01:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:01:49.537472+00:00 (in 2.600050 seconds) securityworker stdout | 2023-01-17 05:01:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:16 UTC)" (scheduled at 2023-01-17 05:01:46.936988+00:00) securityworker stdout | 2023-01-17 05:01:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:01:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:01:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:16.936988+00:00 (in 27.399033 seconds) securityworker stdout | 2023-01-17 05:01:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:19 UTC)" (scheduled at 2023-01-17 05:01:49.537472+00:00) securityworker stdout | 2023-01-17 05:01:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:01:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:01:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:01:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:01:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:01:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:01:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:01:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:01:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:01:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:01:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:01:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:01:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:01:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:01:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:01:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:21.348605+00:00 (in 29.999576 seconds) gcworker stdout | 2023-01-17 05:01:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:21 UTC)" (scheduled at 2023-01-17 05:01:51.348605+00:00) gcworker stdout | 2023-01-17 05:01:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722111349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:01:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:01:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:01:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:01:52,059 [231] [DEBUG] [app] Starting request: urn:request:03edb1ab-da4c-4bb6-8a73-4f2e1fc4e500 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:52,059 [236] [DEBUG] [app] Starting request: urn:request:64494399-2efe-4920-8b93-b03907120b6f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:01:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:01:52,065 [243] [DEBUG] [app] Starting request: urn:request:e9a74551-3e62-4a0f-9fc7-bc2c479c29aa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:52,065 [230] [DEBUG] [app] Starting request: urn:request:d7f73797-92ce-46c4-9de4-67cb9a8e8784 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:01:52,065 [243] [DEBUG] [app] Ending request: urn:request:e9a74551-3e62-4a0f-9fc7-bc2c479c29aa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e9a74551-3e62-4a0f-9fc7-bc2c479c29aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:01:52,066 [230] [DEBUG] [app] Ending request: urn:request:d7f73797-92ce-46c4-9de4-67cb9a8e8784 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d7f73797-92ce-46c4-9de4-67cb9a8e8784', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:01:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:01:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:01:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:01:52,071 [231] [DEBUG] [app] Starting request: urn:request:e5fbdbbc-a30b-402d-a235-2ff225d6332f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:01:52,071 [231] [DEBUG] [app] Ending request: urn:request:e5fbdbbc-a30b-402d-a235-2ff225d6332f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5fbdbbc-a30b-402d-a235-2ff225d6332f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:52,071 [233] [DEBUG] [app] Starting request: urn:request:bc5a6bc3-d004-4fe1-9e3e-8ac227965fb0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:01:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:01:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:52,072 [233] [DEBUG] [app] Ending request: urn:request:bc5a6bc3-d004-4fe1-9e3e-8ac227965fb0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc5a6bc3-d004-4fe1-9e3e-8ac227965fb0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:01:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:01:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:01:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:01:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:01:52,088 [231] [DEBUG] [app] Ending request: urn:request:03edb1ab-da4c-4bb6-8a73-4f2e1fc4e500 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:03edb1ab-da4c-4bb6-8a73-4f2e1fc4e500', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:52,088 [236] [DEBUG] [app] Ending request: urn:request:64494399-2efe-4920-8b93-b03907120b6f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:64494399-2efe-4920-8b93-b03907120b6f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:01:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:01:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:01:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) exportactionlogsworker stdout | 2023-01-17 05:01:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:01:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:28.963662+00:00 (in 33.999807 seconds) exportactionlogsworker stdout | 2023-01-17 05:01:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:54 UTC)" (scheduled at 2023-01-17 05:01:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:01:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 1, 54, 964328), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:01:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 54, 964328), True, datetime.datetime(2023, 1, 17, 5, 1, 54, 964328), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:01:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 1, 54, 964328), True, datetime.datetime(2023, 1, 17, 5, 1, 54, 964328), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 1, 54, 964328), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:01:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:01:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:06:54 UTC)" executed successfully builder stdout | 2023-01-17 05:01:55,640 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue notificationworker stdout | 2023-01-17 05:01:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:01:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:01.636362+00:00 (in 5.989597 seconds) notificationworker stdout | 2023-01-17 05:01:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:05 UTC)" (scheduled at 2023-01-17 05:01:55.646354+00:00) notificationworker stdout | 2023-01-17 05:01:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:01:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 1, 55, 647154), True, datetime.datetime(2023, 1, 17, 5, 1, 55, 647154), 0, 'notification/%', 50, 1, 0]) builder stdout | 2023-01-17 05:01:55,652 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:01:55,652 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics notificationworker stdout | 2023-01-17 05:01:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:01:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:05 UTC)" executed successfully builder stdout | 2023-01-17 05:01:55,663 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:01:57,302 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:01:57,311 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:01:58,216 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:01:58,743 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:01:58,783 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:01:59,126 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:01:59,204 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:02:00,625 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:02:00,629 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:02:00,631 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:02:00,634 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:02:00,636 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:02:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:05.646354+00:00 (in 4.009511 seconds) notificationworker stdout | 2023-01-17 05:02:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:01 UTC)" (scheduled at 2023-01-17 05:02:01.636362+00:00) notificationworker stdout | 2023-01-17 05:02:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:02:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:02:02,295 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:02:02,672 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:02:03,798 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:02:03,859 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:02:04,025 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:02:04,531 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:02:04,630 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:02:04,633 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:02:04,636 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:02:04,757 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:02:04,862 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:02:04,865 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:02:04,867 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:02:04,870 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:02:04,873 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:02:04,875 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:02:04,877 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:02:04,883 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:02:04,885 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} repositorygcworker stdout | 2023-01-17 05:02:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:02:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:19.346120+00:00 (in 14.408847 seconds) repositorygcworker stdout | 2023-01-17 05:02:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:04 UTC)" (scheduled at 2023-01-17 05:02:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:02:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:02:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 4, 937723), True, datetime.datetime(2023, 1, 17, 5, 2, 4, 937723), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:02:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:02:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:02:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:04 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:02:05,009 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:02:05,197 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:02:05,527 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:02:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:15.646354+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:02:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:15 UTC)" (scheduled at 2023-01-17 05:02:05.646354+00:00) notificationworker stdout | 2023-01-17 05:02:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 5, 647302), True, datetime.datetime(2023, 1, 17, 5, 2, 5, 647302), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:15 UTC)" executed successfully builder stdout | 2023-01-17 05:02:05,673 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:05,685 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:05,685 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:05,696 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:02:06,033 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:02:06,218 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:02:06,333 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} namespacegcworker stdout | 2023-01-17 05:02:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:02:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:41.148245+00:00 (in 34.206558 seconds) namespacegcworker stdout | 2023-01-17 05:02:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:07:06 UTC)" (scheduled at 2023-01-17 05:02:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:02:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 2, 6, 942079), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:02:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 2, 6, 942079), True, datetime.datetime(2023, 1, 17, 5, 2, 6, 942079), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:02:06,956 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 2, 6, 942079), True, datetime.datetime(2023, 1, 17, 5, 2, 6, 942079), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 2, 6, 942079), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:02:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:02:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:07:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:02:07,060 [231] [DEBUG] [app] Starting request: urn:request:7d37e7d8-8d49-4bb4-80df-1df5687f0ccd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:07,060 [233] [DEBUG] [app] Starting request: urn:request:fe7b1703-bc29-49c1-bfbc-42dc4a4718b6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:07,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:07,065 [242] [DEBUG] [app] Starting request: urn:request:c33e591f-2bfb-4fcf-9b3d-df23f33bb8c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:07,066 [242] [DEBUG] [app] Ending request: urn:request:c33e591f-2bfb-4fcf-9b3d-df23f33bb8c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c33e591f-2bfb-4fcf-9b3d-df23f33bb8c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:07,066 [230] [DEBUG] [app] Starting request: urn:request:81a81614-cdd8-4c6f-8a01-7738c860c084 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:02:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:02:07,066 [230] [DEBUG] [app] Ending request: urn:request:81a81614-cdd8-4c6f-8a01-7738c860c084 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81a81614-cdd8-4c6f-8a01-7738c860c084', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:02:07,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:07,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:07,070 [236] [DEBUG] [app] Starting request: urn:request:95d422bc-d4a1-4037-8058-50948d3d2b21 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:07,071 [236] [DEBUG] [app] Ending request: urn:request:95d422bc-d4a1-4037-8058-50948d3d2b21 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95d422bc-d4a1-4037-8058-50948d3d2b21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:07,071 [231] [DEBUG] [app] Starting request: urn:request:a02068ca-54fc-45d6-88a3-2d142d218f6d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:02:07,071 [231] [DEBUG] [app] Ending request: urn:request:a02068ca-54fc-45d6-88a3-2d142d218f6d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a02068ca-54fc-45d6-88a3-2d142d218f6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:07,087 [231] [DEBUG] [app] Ending request: urn:request:7d37e7d8-8d49-4bb4-80df-1df5687f0ccd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d37e7d8-8d49-4bb4-80df-1df5687f0ccd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:07,088 [233] [DEBUG] [app] Ending request: urn:request:fe7b1703-bc29-49c1-bfbc-42dc4a4718b6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fe7b1703-bc29-49c1-bfbc-42dc4a4718b6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:02:07,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:02:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 05:02:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:02:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:40.738568+00:00 (in 29.999569 seconds) buildlogsarchiver stdout | 2023-01-17 05:02:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:40 UTC)" (scheduled at 2023-01-17 05:02:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:02:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 2, 10, 739396), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:02:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:02:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:02:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:02:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:25.646354+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 05:02:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:25 UTC)" (scheduled at 2023-01-17 05:02:15.646354+00:00) notificationworker stdout | 2023-01-17 05:02:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 15, 647213), True, datetime.datetime(2023, 1, 17, 5, 2, 15, 647213), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:25 UTC)" executed successfully builder stdout | 2023-01-17 05:02:15,706 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:15,718 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:15,718 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:15,728 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:19.537472+00:00 (in 2.599983 seconds) securityworker stdout | 2023-01-17 05:02:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:46 UTC)" (scheduled at 2023-01-17 05:02:16.936988+00:00) securityworker stdout | 2023-01-17 05:02:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:02:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:02:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:04.936835+00:00 (in 45.590249 seconds) repositorygcworker stdout | 2023-01-17 05:02:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:19 UTC)" (scheduled at 2023-01-17 05:02:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:02:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:02:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:02:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:46.936988+00:00 (in 27.399045 seconds) securityworker stdout | 2023-01-17 05:02:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:49 UTC)" (scheduled at 2023-01-17 05:02:19.537472+00:00) securityworker stdout | 2023-01-17 05:02:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:02:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:02:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:02:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:02:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:02:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:02:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:02:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:02:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:51.348605+00:00 (in 29.999528 seconds) gcworker stdout | 2023-01-17 05:02:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:51 UTC)" (scheduled at 2023-01-17 05:02:21.348605+00:00) gcworker stdout | 2023-01-17 05:02:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722141349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:02:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:02:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:02:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:02:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:02:22,059 [231] [DEBUG] [app] Starting request: urn:request:8c374701-3186-4a7f-94b1-8709809e7538 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:22,060 [235] [DEBUG] [app] Starting request: urn:request:2a2f5a89-ce6f-43d2-b7f4-a8d71325d12c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:22,064 [239] [DEBUG] [app] Starting request: urn:request:09634f67-6af1-41e8-aa75-b7aa973a7434 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:22,065 [239] [DEBUG] [app] Ending request: urn:request:09634f67-6af1-41e8-aa75-b7aa973a7434 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09634f67-6af1-41e8-aa75-b7aa973a7434', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:22,065 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:02:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:02:22,066 [237] [DEBUG] [app] Starting request: urn:request:81e199d9-1546-4fab-a5d2-b641bedef403 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-registry stdout | 2023-01-17 05:02:22,067 [237] [DEBUG] [app] Ending request: urn:request:81e199d9-1546-4fab-a5d2-b641bedef403 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81e199d9-1546-4fab-a5d2-b641bedef403', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:22,067 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.004 162 0.004) gunicorn-web stdout | 2023-01-17 05:02:22,068 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:22,069 [233] [DEBUG] [app] Starting request: urn:request:0879148f-20ae-4111-b30d-35bd4be69c54 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:22,070 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:22,070 [233] [DEBUG] [app] Ending request: urn:request:0879148f-20ae-4111-b30d-35bd4be69c54 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0879148f-20ae-4111-b30d-35bd4be69c54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:02:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:22,072 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:22,072 [233] [DEBUG] [app] Starting request: urn:request:93d0d208-47af-4962-86aa-2ff1ea559d62 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:22,072 [233] [DEBUG] [app] Ending request: urn:request:93d0d208-47af-4962-86aa-2ff1ea559d62 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93d0d208-47af-4962-86aa-2ff1ea559d62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:22,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:22,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:22,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:22,087 [231] [DEBUG] [app] Ending request: urn:request:8c374701-3186-4a7f-94b1-8709809e7538 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8c374701-3186-4a7f-94b1-8709809e7538', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:02:22,089 [235] [DEBUG] [app] Ending request: urn:request:2a2f5a89-ce6f-43d2-b7f4-a8d71325d12c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a2f5a89-ce6f-43d2-b7f4-a8d71325d12c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) notificationworker stdout | 2023-01-17 05:02:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:35.646354+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 05:02:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:35 UTC)" (scheduled at 2023-01-17 05:02:25.646354+00:00) notificationworker stdout | 2023-01-17 05:02:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 25, 647235), True, datetime.datetime(2023, 1, 17, 5, 2, 25, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:35 UTC)" executed successfully builder stdout | 2023-01-17 05:02:25,739 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:25,751 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:25,751 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:25,761 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:27,337 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:02:27,346 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:02:28,238 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:02:28,778 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:02:28,807 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} exportactionlogsworker stdout | 2023-01-17 05:02:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:02:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:35.853396+00:00 (in 6.889178 seconds) exportactionlogsworker stdout | 2023-01-17 05:02:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:28 UTC)" (scheduled at 2023-01-17 05:02:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:02:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:02:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:28 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:02:29,156 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:02:29,238 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:02:30,632 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:02:30,634 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:02:30,638 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:02:30,641 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:02:30,643 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:02:32,330 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:02:32,708 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:02:33,810 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:02:33,886 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:02:34,049 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:02:34,567 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:02:34,637 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:02:34,640 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:02:34,642 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:02:34,792 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:02:34,872 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:02:34,875 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:02:34,879 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:02:34,881 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:02:34,884 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:02:34,886 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:02:34,888 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:02:34,891 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:02:34,894 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} securityscanningnotificationworker stdout | 2023-01-17 05:02:35,045 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:02:35,230 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:02:35,562 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:02:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:45.646354+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:02:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:45 UTC)" (scheduled at 2023-01-17 05:02:35.646354+00:00) notificationworker stdout | 2023-01-17 05:02:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 35, 647241), True, datetime.datetime(2023, 1, 17, 5, 2, 35, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:45 UTC)" executed successfully builder stdout | 2023-01-17 05:02:35,771 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:35,783 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:35,783 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:35,794 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:02:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:02:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:28.963662+00:00 (in 53.109767 seconds) exportactionlogsworker stdout | 2023-01-17 05:02:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:35 UTC)" (scheduled at 2023-01-17 05:02:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:02:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:02:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 35, 854337), True, datetime.datetime(2023, 1, 17, 5, 2, 35, 854337), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:02:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:02:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:02:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:02:36,068 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:02:36,253 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:02:36,368 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:02:37,060 [236] [DEBUG] [app] Starting request: urn:request:2737ad1f-e2fe-458c-933d-1f9edf49bb60 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:37,060 [233] [DEBUG] [app] Starting request: urn:request:b78b4346-df34-4cd0-8c6a-bbeb685a4d6b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:37,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:37,065 [243] [DEBUG] [app] Starting request: urn:request:5cc9464b-a99b-471e-8b91-6642c2708f97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:37,065 [243] [DEBUG] [app] Ending request: urn:request:5cc9464b-a99b-471e-8b91-6642c2708f97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cc9464b-a99b-471e-8b91-6642c2708f97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:37,065 [230] [DEBUG] [app] Starting request: urn:request:e173f4ab-109c-4000-97e5-3d13d2fae4e0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:02:37,066 [230] [DEBUG] [app] Ending request: urn:request:e173f4ab-109c-4000-97e5-3d13d2fae4e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e173f4ab-109c-4000-97e5-3d13d2fae4e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:02:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.003) gunicorn-web stdout | 2023-01-17 05:02:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:37,071 [236] [DEBUG] [app] Starting request: urn:request:11ca39e1-367c-46a9-8ea9-7bfd5b88608a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:37,071 [231] [DEBUG] [app] Starting request: urn:request:ec043b2e-01f6-4435-8146-38a0d6af24e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:37,072 [236] [DEBUG] [app] Ending request: urn:request:11ca39e1-367c-46a9-8ea9-7bfd5b88608a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11ca39e1-367c-46a9-8ea9-7bfd5b88608a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:37,072 [231] [DEBUG] [app] Ending request: urn:request:ec043b2e-01f6-4435-8146-38a0d6af24e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec043b2e-01f6-4435-8146-38a0d6af24e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:37,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:37,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:37,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:37,088 [236] [DEBUG] [app] Ending request: urn:request:2737ad1f-e2fe-458c-933d-1f9edf49bb60 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2737ad1f-e2fe-458c-933d-1f9edf49bb60', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:37,088 [233] [DEBUG] [app] Ending request: urn:request:b78b4346-df34-4cd0-8c6a-bbeb685a4d6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b78b4346-df34-4cd0-8c6a-bbeb685a4d6b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:37,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:02:37,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:02:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:02:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:37.537407+00:00 (in 0.280312 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:02:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:37 UTC)" (scheduled at 2023-01-17 05:02:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:02:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:02:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 37, 257490), True, datetime.datetime(2023, 1, 17, 5, 2, 37, 257490), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:02:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:02:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:02:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:02:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:02:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:37.256652+00:00 (in 59.718826 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:02:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:37 UTC)" (scheduled at 2023-01-17 05:02:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:02:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:02:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:02:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:02:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:10.738568+00:00 (in 29.999524 seconds) buildlogsarchiver stdout | 2023-01-17 05:02:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:10 UTC)" (scheduled at 2023-01-17 05:02:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:02:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 2, 40, 739515), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:02:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:02:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:02:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:02:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:02:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:42.941536+00:00 (in 1.792798 seconds) namespacegcworker stdout | 2023-01-17 05:02:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:41 UTC)" (scheduled at 2023-01-17 05:02:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:02:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:02:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 41, 149180), True, datetime.datetime(2023, 1, 17, 5, 2, 41, 149180), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:02:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:02:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:02:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:02:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:02:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:41.148245+00:00 (in 58.206233 seconds) namespacegcworker stdout | 2023-01-17 05:02:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:42 UTC)" (scheduled at 2023-01-17 05:02:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:02:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:02:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:03:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:02:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:55.646354+00:00 (in 9.999486 seconds) notificationworker stdout | 2023-01-17 05:02:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:55 UTC)" (scheduled at 2023-01-17 05:02:45.646354+00:00) notificationworker stdout | 2023-01-17 05:02:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 45, 647295), True, datetime.datetime(2023, 1, 17, 5, 2, 45, 647295), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:02:55 UTC)" executed successfully builder stdout | 2023-01-17 05:02:45,795 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:45,807 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:45,807 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:45,818 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:02:49.537472+00:00 (in 2.600030 seconds) securityworker stdout | 2023-01-17 05:02:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:16 UTC)" (scheduled at 2023-01-17 05:02:46.936988+00:00) securityworker stdout | 2023-01-17 05:02:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:02:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:02:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:16.936988+00:00 (in 27.399039 seconds) securityworker stdout | 2023-01-17 05:02:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:19 UTC)" (scheduled at 2023-01-17 05:02:49.537472+00:00) securityworker stdout | 2023-01-17 05:02:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:02:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:02:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:02:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:02:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:02:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:02:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:02:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:02:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:02:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:02:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:02:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:02:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:02:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:02:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:02:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:21.348605+00:00 (in 29.999568 seconds) gcworker stdout | 2023-01-17 05:02:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:21 UTC)" (scheduled at 2023-01-17 05:02:51.348605+00:00) gcworker stdout | 2023-01-17 05:02:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722171349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:02:51,364 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:02:51,364 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:02:51,364 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:02:52,060 [231] [DEBUG] [app] Starting request: urn:request:857db66c-ed63-4ade-a121-f6403294b323 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:52,060 [236] [DEBUG] [app] Starting request: urn:request:7d947c96-d504-4e14-bb31-632487700442 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:02:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:02:52,065 [230] [DEBUG] [app] Starting request: urn:request:d84891bb-813b-4796-a4b6-9975f7063156 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,065 [243] [DEBUG] [app] Starting request: urn:request:f4af1d40-e576-4826-a120-9c6a4984325b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,065 [230] [DEBUG] [app] Ending request: urn:request:d84891bb-813b-4796-a4b6-9975f7063156 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d84891bb-813b-4796-a4b6-9975f7063156', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,065 [243] [DEBUG] [app] Ending request: urn:request:f4af1d40-e576-4826-a120-9c6a4984325b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4af1d40-e576-4826-a120-9c6a4984325b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:02:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:02:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:02:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:02:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:02:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:02:52,070 [231] [DEBUG] [app] Starting request: urn:request:3c4a4bb9-c75e-4a0b-916a-0711210f479e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:52,070 [235] [DEBUG] [app] Starting request: urn:request:514d46de-6157-4723-b016-d6e6afafea2f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:02:52,070 [231] [DEBUG] [app] Ending request: urn:request:3c4a4bb9-c75e-4a0b-916a-0711210f479e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c4a4bb9-c75e-4a0b-916a-0711210f479e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:52,071 [235] [DEBUG] [app] Ending request: urn:request:514d46de-6157-4723-b016-d6e6afafea2f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:514d46de-6157-4723-b016-d6e6afafea2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:02:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:02:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:02:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:02:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:02:52,087 [236] [DEBUG] [app] Ending request: urn:request:7d947c96-d504-4e14-bb31-632487700442 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d947c96-d504-4e14-bb31-632487700442', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:52,087 [231] [DEBUG] [app] Ending request: urn:request:857db66c-ed63-4ade-a121-f6403294b323 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:857db66c-ed63-4ade-a121-f6403294b323', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:02:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:02:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:02:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) notificationworker stdout | 2023-01-17 05:02:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:02:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:01.636362+00:00 (in 5.989504 seconds) notificationworker stdout | 2023-01-17 05:02:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:05 UTC)" (scheduled at 2023-01-17 05:02:55.646354+00:00) notificationworker stdout | 2023-01-17 05:02:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:02:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 2, 55, 647284), True, datetime.datetime(2023, 1, 17, 5, 2, 55, 647284), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:02:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:02:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:02:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:05 UTC)" executed successfully builder stdout | 2023-01-17 05:02:55,828 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:02:55,840 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:02:55,840 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:02:55,851 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:02:57,365 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:02:57,371 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:02:58,255 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:02:58,812 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:02:58,831 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:02:59,182 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:02:59,274 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:03:00,639 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:03:00,641 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:03:00,643 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:03:00,647 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:03:00,649 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:03:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:05.646354+00:00 (in 4.009514 seconds) notificationworker stdout | 2023-01-17 05:03:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:01 UTC)" (scheduled at 2023-01-17 05:03:01.636362+00:00) notificationworker stdout | 2023-01-17 05:03:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:03:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:03:02,352 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:03:02,714 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:03:03,840 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:03:03,921 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:03:04,055 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:03:04,590 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:03:04,644 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:03:04,646 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:03:04,649 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:03:04,827 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:03:04,882 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:03:04,887 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:03:04,890 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:03:04,892 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:03:04,895 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:03:04,897 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:03:04,899 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:03:04,902 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:03:04,904 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} repositorygcworker stdout | 2023-01-17 05:03:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:03:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:19.346120+00:00 (in 14.408858 seconds) repositorygcworker stdout | 2023-01-17 05:03:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:04 UTC)" (scheduled at 2023-01-17 05:03:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:03:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:03:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 4, 937717), True, datetime.datetime(2023, 1, 17, 5, 3, 4, 937717), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:03:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:03:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:03:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:04 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:03:05,079 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:03:05,260 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:03:05,596 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:03:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:15.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:03:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:15 UTC)" (scheduled at 2023-01-17 05:03:05.646354+00:00) notificationworker stdout | 2023-01-17 05:03:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 5, 647184), True, datetime.datetime(2023, 1, 17, 5, 3, 5, 647184), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:15 UTC)" executed successfully builder stdout | 2023-01-17 05:03:05,861 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:05,872 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:05,873 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:05,883 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:03:06,100 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:03:06,262 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:03:06,386 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:03:07,060 [231] [DEBUG] [app] Starting request: urn:request:8630f60c-f498-43f2-8031-9a9ee5bd9e6e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:07,060 [236] [DEBUG] [app] Starting request: urn:request:93a49a58-732b-46df-96f7-e65d6be3796f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:07,065 [230] [DEBUG] [app] Starting request: urn:request:032175b9-d6da-4bcb-ba0e-c69ff53f3fdf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,065 [242] [DEBUG] [app] Starting request: urn:request:58618c3b-37e1-44a5-9730-ccb600295482 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,066 [230] [DEBUG] [app] Ending request: urn:request:032175b9-d6da-4bcb-ba0e-c69ff53f3fdf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:032175b9-d6da-4bcb-ba0e-c69ff53f3fdf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,066 [242] [DEBUG] [app] Ending request: urn:request:58618c3b-37e1-44a5-9730-ccb600295482 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58618c3b-37e1-44a5-9730-ccb600295482', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:03:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:03:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:07,071 [235] [DEBUG] [app] Starting request: urn:request:def18ab7-ce49-433b-8cb0-395a700dca2c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:07,071 [231] [DEBUG] [app] Starting request: urn:request:6e75cde7-2e23-4b14-bfd6-66b1d8e4f4b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:07,071 [235] [DEBUG] [app] Ending request: urn:request:def18ab7-ce49-433b-8cb0-395a700dca2c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:def18ab7-ce49-433b-8cb0-395a700dca2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:07,071 [231] [DEBUG] [app] Ending request: urn:request:6e75cde7-2e23-4b14-bfd6-66b1d8e4f4b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e75cde7-2e23-4b14-bfd6-66b1d8e4f4b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:07,088 [231] [DEBUG] [app] Ending request: urn:request:8630f60c-f498-43f2-8031-9a9ee5bd9e6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8630f60c-f498-43f2-8031-9a9ee5bd9e6e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:07,088 [236] [DEBUG] [app] Ending request: urn:request:93a49a58-732b-46df-96f7-e65d6be3796f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93a49a58-732b-46df-96f7-e65d6be3796f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:03:07,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:03:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 05:03:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:03:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:40.738568+00:00 (in 29.999416 seconds) buildlogsarchiver stdout | 2023-01-17 05:03:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:40 UTC)" (scheduled at 2023-01-17 05:03:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:03:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 3, 10, 739631), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:03:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:03:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:03:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:03:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:25.646354+00:00 (in 9.999583 seconds) notificationworker stdout | 2023-01-17 05:03:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:25 UTC)" (scheduled at 2023-01-17 05:03:15.646354+00:00) notificationworker stdout | 2023-01-17 05:03:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 15, 647180), True, datetime.datetime(2023, 1, 17, 5, 3, 15, 647180), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:25 UTC)" executed successfully builder stdout | 2023-01-17 05:03:15,883 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:15,895 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:15,895 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:15,906 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:19.537472+00:00 (in 2.600028 seconds) securityworker stdout | 2023-01-17 05:03:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:46 UTC)" (scheduled at 2023-01-17 05:03:16.936988+00:00) securityworker stdout | 2023-01-17 05:03:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:03:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:03:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:04.936835+00:00 (in 45.590198 seconds) repositorygcworker stdout | 2023-01-17 05:03:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:19 UTC)" (scheduled at 2023-01-17 05:03:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:03:19,347 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:03:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:03:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:46.936988+00:00 (in 27.399074 seconds) securityworker stdout | 2023-01-17 05:03:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:49 UTC)" (scheduled at 2023-01-17 05:03:19.537472+00:00) securityworker stdout | 2023-01-17 05:03:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:03:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:03:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:03:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:03:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:03:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:03:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:03:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:03:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:51.348605+00:00 (in 29.999589 seconds) gcworker stdout | 2023-01-17 05:03:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:51 UTC)" (scheduled at 2023-01-17 05:03:21.348605+00:00) gcworker stdout | 2023-01-17 05:03:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722201349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:03:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:03:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:03:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:03:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:03:22,060 [236] [DEBUG] [app] Starting request: urn:request:72a9539e-fc45-4b4e-8751-7cc8846dc6aa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:22,060 [233] [DEBUG] [app] Starting request: urn:request:b8f07803-d17c-4070-ade8-cb5971ec5d82 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:22,065 [242] [DEBUG] [app] Starting request: urn:request:f7f6f08f-afec-47a7-a0d0-161982b30680 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:22,066 [240] [DEBUG] [app] Starting request: urn:request:37dd3eaa-b2e6-4f9c-9e5b-aaa597903195 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:22,066 [242] [DEBUG] [app] Ending request: urn:request:f7f6f08f-afec-47a7-a0d0-161982b30680 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f7f6f08f-afec-47a7-a0d0-161982b30680', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:03:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:03:22,066 [240] [DEBUG] [app] Ending request: urn:request:37dd3eaa-b2e6-4f9c-9e5b-aaa597903195 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:37dd3eaa-b2e6-4f9c-9e5b-aaa597903195', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:03:22,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:03:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:22,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:22,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:22,071 [235] [DEBUG] [app] Starting request: urn:request:f45b51b9-c9e4-4e73-816f-bf40d693be91 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:22,071 [231] [DEBUG] [app] Starting request: urn:request:b2533716-6be8-42b9-8c50-b1c50eb605ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:22,071 [235] [DEBUG] [app] Ending request: urn:request:f45b51b9-c9e4-4e73-816f-bf40d693be91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f45b51b9-c9e4-4e73-816f-bf40d693be91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:22,072 [231] [DEBUG] [app] Ending request: urn:request:b2533716-6be8-42b9-8c50-b1c50eb605ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2533716-6be8-42b9-8c50-b1c50eb605ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:03:22,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:22,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:22,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:22,088 [233] [DEBUG] [app] Ending request: urn:request:b8f07803-d17c-4070-ade8-cb5971ec5d82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b8f07803-d17c-4070-ade8-cb5971ec5d82', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:22,088 [236] [DEBUG] [app] Ending request: urn:request:72a9539e-fc45-4b4e-8751-7cc8846dc6aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:72a9539e-fc45-4b4e-8751-7cc8846dc6aa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:22,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:03:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) notificationworker stdout | 2023-01-17 05:03:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:35.646354+00:00 (in 9.999508 seconds) notificationworker stdout | 2023-01-17 05:03:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:35 UTC)" (scheduled at 2023-01-17 05:03:25.646354+00:00) notificationworker stdout | 2023-01-17 05:03:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 25, 647243), True, datetime.datetime(2023, 1, 17, 5, 3, 25, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:35 UTC)" executed successfully builder stdout | 2023-01-17 05:03:25,912 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:25,924 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:25,924 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:25,935 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:03:27,372 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:03:27,406 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:03:28,283 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:03:28,849 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:03:28,851 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} exportactionlogsworker stdout | 2023-01-17 05:03:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:03:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:35.853396+00:00 (in 6.889282 seconds) exportactionlogsworker stdout | 2023-01-17 05:03:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:28 UTC)" (scheduled at 2023-01-17 05:03:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:03:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:03:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:28 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:03:29,205 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:03:29,309 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:03:30,646 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:03:30,649 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:03:30,651 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:03:30,654 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:03:30,656 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:03:32,374 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:03:32,739 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:03:33,847 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:03:33,931 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:03:34,089 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:03:34,615 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:03:34,651 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:03:34,653 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:03:34,655 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:03:34,856 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:03:34,891 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:03:34,895 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:03:34,898 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:03:34,901 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:03:34,903 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:03:34,905 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:03:34,909 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:03:34,911 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:03:34,913 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} securityscanningnotificationworker stdout | 2023-01-17 05:03:35,100 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:03:35,269 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:03:35,631 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:03:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:36.635997+00:00 (in 0.989204 seconds) notificationworker stdout | 2023-01-17 05:03:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:45 UTC)" (scheduled at 2023-01-17 05:03:35.646354+00:00) notificationworker stdout | 2023-01-17 05:03:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 35, 647168), True, datetime.datetime(2023, 1, 17, 5, 3, 35, 647168), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:03:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:03:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:28.963662+00:00 (in 53.109760 seconds) exportactionlogsworker stdout | 2023-01-17 05:03:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:35 UTC)" (scheduled at 2023-01-17 05:03:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:03:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:03:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 35, 854375), True, datetime.datetime(2023, 1, 17, 5, 3, 35, 854375), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:03:35,871 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:03:35,871 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:03:35,871 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:35 UTC)" executed successfully builder stdout | 2023-01-17 05:03:35,945 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:35,957 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:35,957 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:35,968 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:03:36,136 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:03:36,283 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:03:36,406 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} notificationworker stdout | 2023-01-17 05:03:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:45.646354+00:00 (in 9.009905 seconds) notificationworker stdout | 2023-01-17 05:03:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:08:36 UTC)" (scheduled at 2023-01-17 05:03:36.635997+00:00) notificationworker stdout | 2023-01-17 05:03:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 3, 36, 636821), 'notification/%']) notificationworker stdout | 2023-01-17 05:03:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 3, 36, 636821), True, datetime.datetime(2023, 1, 17, 5, 3, 36, 636821), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:03:36,654 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 3, 36, 636821), True, datetime.datetime(2023, 1, 17, 5, 3, 36, 636821), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 3, 36, 636821), 'notification/%']) notificationworker stdout | 2023-01-17 05:03:36,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:36,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:08:36 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:03:37,060 [235] [DEBUG] [app] Starting request: urn:request:ab7f37c1-23da-4710-90bd-2dc144d76381 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:37,060 [231] [DEBUG] [app] Starting request: urn:request:2062515d-5d73-4e0f-8050-cc6eb59555d9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:37,065 [230] [DEBUG] [app] Starting request: urn:request:3dde9585-7359-423a-8221-dc2c81015ba4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:37,065 [243] [DEBUG] [app] Starting request: urn:request:38c96cf5-3386-47a3-90b2-9035c80d2741 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:37,065 [230] [DEBUG] [app] Ending request: urn:request:3dde9585-7359-423a-8221-dc2c81015ba4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3dde9585-7359-423a-8221-dc2c81015ba4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:37,065 [243] [DEBUG] [app] Ending request: urn:request:38c96cf5-3386-47a3-90b2-9035c80d2741 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:38c96cf5-3386-47a3-90b2-9035c80d2741', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:03:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:37,070 [236] [DEBUG] [app] Starting request: urn:request:fc04dd11-69a5-4422-8c16-8b7930983d69 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:37,070 [233] [DEBUG] [app] Starting request: urn:request:c6c334c4-39c3-47a4-b6ff-cbac6c0c1110 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:37,070 [236] [DEBUG] [app] Ending request: urn:request:fc04dd11-69a5-4422-8c16-8b7930983d69 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc04dd11-69a5-4422-8c16-8b7930983d69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:37,070 [233] [DEBUG] [app] Ending request: urn:request:c6c334c4-39c3-47a4-b6ff-cbac6c0c1110 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6c334c4-39c3-47a4-b6ff-cbac6c0c1110', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:37,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:37,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:37,086 [235] [DEBUG] [app] Ending request: urn:request:ab7f37c1-23da-4710-90bd-2dc144d76381 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab7f37c1-23da-4710-90bd-2dc144d76381', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:37,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:37,087 [231] [DEBUG] [app] Ending request: urn:request:2062515d-5d73-4e0f-8050-cc6eb59555d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2062515d-5d73-4e0f-8050-cc6eb59555d9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:03:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) securityscanningnotificationworker stdout | 2023-01-17 05:03:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:03:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:37.537407+00:00 (in 0.280345 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:03:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:37 UTC)" (scheduled at 2023-01-17 05:03:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:03:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:03:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 37, 257496), True, datetime.datetime(2023, 1, 17, 5, 3, 37, 257496), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:03:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:03:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:03:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:03:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:03:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:37.256652+00:00 (in 59.718850 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:03:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:37 UTC)" (scheduled at 2023-01-17 05:03:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:03:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:03:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:03:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:03:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:10.738568+00:00 (in 29.999508 seconds) buildlogsarchiver stdout | 2023-01-17 05:03:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:10 UTC)" (scheduled at 2023-01-17 05:03:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:03:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 3, 40, 739451), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:03:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:03:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:03:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:03:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:03:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:42.941536+00:00 (in 1.792862 seconds) namespacegcworker stdout | 2023-01-17 05:03:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:41 UTC)" (scheduled at 2023-01-17 05:03:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:03:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:03:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 41, 149044), True, datetime.datetime(2023, 1, 17, 5, 3, 41, 149044), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:03:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:03:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:03:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:03:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:03:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:41.148245+00:00 (in 58.206256 seconds) namespacegcworker stdout | 2023-01-17 05:03:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:42 UTC)" (scheduled at 2023-01-17 05:03:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:03:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:03:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:04:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:03:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:55.646354+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 05:03:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:55 UTC)" (scheduled at 2023-01-17 05:03:45.646354+00:00) notificationworker stdout | 2023-01-17 05:03:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 45, 647218), True, datetime.datetime(2023, 1, 17, 5, 3, 45, 647218), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:03:55 UTC)" executed successfully builder stdout | 2023-01-17 05:03:45,978 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:45,990 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:45,990 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:46,001 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:03:49.537472+00:00 (in 2.599978 seconds) securityworker stdout | 2023-01-17 05:03:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:16 UTC)" (scheduled at 2023-01-17 05:03:46.936988+00:00) securityworker stdout | 2023-01-17 05:03:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:03:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:03:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:16.936988+00:00 (in 27.399038 seconds) securityworker stdout | 2023-01-17 05:03:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:19 UTC)" (scheduled at 2023-01-17 05:03:49.537472+00:00) securityworker stdout | 2023-01-17 05:03:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:03:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:03:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:03:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:03:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:03:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:03:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:03:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:03:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:03:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:03:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:03:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:03:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:03:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:03:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:03:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:21.348605+00:00 (in 29.999598 seconds) gcworker stdout | 2023-01-17 05:03:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:21 UTC)" (scheduled at 2023-01-17 05:03:51.348605+00:00) gcworker stdout | 2023-01-17 05:03:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722231349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:03:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:03:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:03:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:03:52,060 [236] [DEBUG] [app] Starting request: urn:request:cb02d196-544e-4643-8b05-c1dc40ebd712 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:52,060 [233] [DEBUG] [app] Starting request: urn:request:0182b036-975d-4cc9-bb52-e5012c74b254 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:03:52,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:03:52,065 [243] [DEBUG] [app] Starting request: urn:request:d4b354fe-2317-4f86-98a6-c96042df851f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:52,066 [230] [DEBUG] [app] Starting request: urn:request:85ffc699-53c6-4408-9a5c-044113b56ec5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:03:52,066 [243] [DEBUG] [app] Ending request: urn:request:d4b354fe-2317-4f86-98a6-c96042df851f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d4b354fe-2317-4f86-98a6-c96042df851f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:03:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:03:52,066 [230] [DEBUG] [app] Ending request: urn:request:85ffc699-53c6-4408-9a5c-044113b56ec5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85ffc699-53c6-4408-9a5c-044113b56ec5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:03:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:03:52,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:52,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:03:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:03:52,070 [231] [DEBUG] [app] Starting request: urn:request:ccbf2b35-f93b-4a45-9f16-3edb41f370b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:52,071 [233] [DEBUG] [app] Starting request: urn:request:a3ad3c4d-fea3-429e-a095-f94998fb3cd8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:03:52,071 [231] [DEBUG] [app] Ending request: urn:request:ccbf2b35-f93b-4a45-9f16-3edb41f370b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ccbf2b35-f93b-4a45-9f16-3edb41f370b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:03:52,071 [233] [DEBUG] [app] Ending request: urn:request:a3ad3c4d-fea3-429e-a095-f94998fb3cd8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a3ad3c4d-fea3-429e-a095-f94998fb3cd8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:03:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:03:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:03:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:03:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:52,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:03:52,088 [233] [DEBUG] [app] Ending request: urn:request:0182b036-975d-4cc9-bb52-e5012c74b254 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0182b036-975d-4cc9-bb52-e5012c74b254', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:03:52,088 [236] [DEBUG] [app] Ending request: urn:request:cb02d196-544e-4643-8b05-c1dc40ebd712 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb02d196-544e-4643-8b05-c1dc40ebd712', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:03:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:03:52,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:03:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:03:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:03:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:01.636362+00:00 (in 5.989526 seconds) notificationworker stdout | 2023-01-17 05:03:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:05 UTC)" (scheduled at 2023-01-17 05:03:55.646354+00:00) notificationworker stdout | 2023-01-17 05:03:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:03:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 3, 55, 647237), True, datetime.datetime(2023, 1, 17, 5, 3, 55, 647237), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:03:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:03:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:03:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:05 UTC)" executed successfully builder stdout | 2023-01-17 05:03:56,011 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:03:56,024 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:03:56,024 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:03:56,034 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:03:57,408 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:03:57,441 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:03:58,311 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:03:58,870 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:03:58,879 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:03:59,240 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:03:59,345 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:04:00,654 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:04:00,657 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:04:00,659 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:04:00,662 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:04:00,664 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:04:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:05.646354+00:00 (in 4.009484 seconds) notificationworker stdout | 2023-01-17 05:04:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:01 UTC)" (scheduled at 2023-01-17 05:04:01.636362+00:00) notificationworker stdout | 2023-01-17 05:04:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:04:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:04:02,410 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:04:02,775 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:04:03,867 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:04:03,960 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:04:04,097 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:04:04,651 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-secscan stdout | 2023-01-17 05:04:04,657 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:04:04,660 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:04:04,662 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} manifestbackfillworker stdout | 2023-01-17 05:04:04,886 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:04:04,900 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:04:04,903 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:04:04,905 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:04:04,908 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:04:04,910 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:04:04,913 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:04:04,917 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:04:04,919 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:04:04,921 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} repositorygcworker stdout | 2023-01-17 05:04:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:04:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:19.346120+00:00 (in 14.408834 seconds) repositorygcworker stdout | 2023-01-17 05:04:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:04 UTC)" (scheduled at 2023-01-17 05:04:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:04:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:04:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 4, 937667), True, datetime.datetime(2023, 1, 17, 5, 4, 4, 937667), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:04:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:04:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:04:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:04 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:04:05,135 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:04:05,304 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:04:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 05:04:05,646 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:04:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:15.646354+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 05:04:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:15 UTC)" (scheduled at 2023-01-17 05:04:05.646354+00:00) notificationworker stdout | 2023-01-17 05:04:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 5, 647214), True, datetime.datetime(2023, 1, 17, 5, 4, 5, 647214), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:15 UTC)" executed successfully builder stdout | 2023-01-17 05:04:06,045 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:06,056 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:06,057 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:06,067 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:04:06,171 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:04:06,300 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:04:06,440 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:04:07,059 [235] [DEBUG] [app] Starting request: urn:request:d2b92cfc-8846-475e-8118-6bb6fbf73f82 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:07,060 [236] [DEBUG] [app] Starting request: urn:request:12252a49-542a-4afb-af74-a9e8ec85fdc4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:07,064 [243] [DEBUG] [app] Starting request: urn:request:19e987e7-b38f-40c4-9d76-be4ac69ebea7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:07,065 [230] [DEBUG] [app] Starting request: urn:request:c953104a-a2d7-4524-9e3d-986d81685721 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:07,065 [243] [DEBUG] [app] Ending request: urn:request:19e987e7-b38f-40c4-9d76-be4ac69ebea7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:19e987e7-b38f-40c4-9d76-be4ac69ebea7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:07,065 [230] [DEBUG] [app] Ending request: urn:request:c953104a-a2d7-4524-9e3d-986d81685721 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c953104a-a2d7-4524-9e3d-986d81685721', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:04:07,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:04:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:07,069 [233] [DEBUG] [app] Starting request: urn:request:c1bd7083-f12a-4839-abf9-de0c69ddcba3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:07,070 [231] [DEBUG] [app] Starting request: urn:request:15462b08-4ff7-430a-955b-2d524cb07e79 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:07,070 [233] [DEBUG] [app] Ending request: urn:request:c1bd7083-f12a-4839-abf9-de0c69ddcba3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c1bd7083-f12a-4839-abf9-de0c69ddcba3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:07,070 [231] [DEBUG] [app] Ending request: urn:request:15462b08-4ff7-430a-955b-2d524cb07e79 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:15462b08-4ff7-430a-955b-2d524cb07e79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:07,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:04:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:07,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:07,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:07,086 [235] [DEBUG] [app] Ending request: urn:request:d2b92cfc-8846-475e-8118-6bb6fbf73f82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d2b92cfc-8846-475e-8118-6bb6fbf73f82', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:07,086 [236] [DEBUG] [app] Ending request: urn:request:12252a49-542a-4afb-af74-a9e8ec85fdc4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:12252a49-542a-4afb-af74-a9e8ec85fdc4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:07,087 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:04:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:04:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 05:04:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:04:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:40.738568+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 05:04:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:40 UTC)" (scheduled at 2023-01-17 05:04:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:04:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 4, 10, 739431), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:04:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:04:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:04:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:04:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:25.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:04:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:25 UTC)" (scheduled at 2023-01-17 05:04:15.646354+00:00) notificationworker stdout | 2023-01-17 05:04:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 15, 647198), True, datetime.datetime(2023, 1, 17, 5, 4, 15, 647198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:25 UTC)" executed successfully builder stdout | 2023-01-17 05:04:16,068 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:16,080 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:16,080 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:16,091 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:19.537472+00:00 (in 2.599988 seconds) securityworker stdout | 2023-01-17 05:04:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:46 UTC)" (scheduled at 2023-01-17 05:04:16.936988+00:00) securityworker stdout | 2023-01-17 05:04:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:04:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:04:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:04.936835+00:00 (in 45.590239 seconds) repositorygcworker stdout | 2023-01-17 05:04:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:19 UTC)" (scheduled at 2023-01-17 05:04:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:04:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:04:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:04:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:46.936988+00:00 (in 27.399022 seconds) securityworker stdout | 2023-01-17 05:04:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:49 UTC)" (scheduled at 2023-01-17 05:04:19.537472+00:00) securityworker stdout | 2023-01-17 05:04:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:04:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:04:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:04:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:04:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:04:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:04:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:04:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:04:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:51.348605+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 05:04:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:51 UTC)" (scheduled at 2023-01-17 05:04:21.348605+00:00) gcworker stdout | 2023-01-17 05:04:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722261349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:04:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:04:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:04:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:04:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:04:22,059 [231] [DEBUG] [app] Starting request: urn:request:a6e557d4-a24f-40aa-ae2c-f9750d21db3a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:22,059 [236] [DEBUG] [app] Starting request: urn:request:00d714b2-f8de-465f-a586-d44d96f26927 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:22,064 [230] [DEBUG] [app] Starting request: urn:request:10ce34e8-ecf3-45aa-a843-3955d66b8c39 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:22,064 [243] [DEBUG] [app] Starting request: urn:request:04969d67-0a13-4a8e-98fc-1ae2776097b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:22,064 [230] [DEBUG] [app] Ending request: urn:request:10ce34e8-ecf3-45aa-a843-3955d66b8c39 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:10ce34e8-ecf3-45aa-a843-3955d66b8c39', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:22,065 [243] [DEBUG] [app] Ending request: urn:request:04969d67-0a13-4a8e-98fc-1ae2776097b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04969d67-0a13-4a8e-98fc-1ae2776097b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:04:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:04:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:22,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:22,069 [235] [DEBUG] [app] Starting request: urn:request:349d4e76-ad5c-4897-b1c0-9d67edda88f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:22,069 [231] [DEBUG] [app] Starting request: urn:request:485db2c4-b4a8-401c-8ab4-3c3c95127a08 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:22,070 [235] [DEBUG] [app] Ending request: urn:request:349d4e76-ad5c-4897-b1c0-9d67edda88f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:349d4e76-ad5c-4897-b1c0-9d67edda88f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:22,070 [231] [DEBUG] [app] Ending request: urn:request:485db2c4-b4a8-401c-8ab4-3c3c95127a08 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:485db2c4-b4a8-401c-8ab4-3c3c95127a08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:04:22,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:22,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:22,089 [236] [DEBUG] [app] Ending request: urn:request:00d714b2-f8de-465f-a586-d44d96f26927 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:00d714b2-f8de-465f-a586-d44d96f26927', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) gunicorn-web stdout | 2023-01-17 05:04:22,090 [231] [DEBUG] [app] Ending request: urn:request:a6e557d4-a24f-40aa-ae2c-f9750d21db3a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6e557d4-a24f-40aa-ae2c-f9750d21db3a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:22,090 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) gunicorn-web stdout | 2023-01-17 05:04:22,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:04:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:35.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:04:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:35 UTC)" (scheduled at 2023-01-17 05:04:25.646354+00:00) notificationworker stdout | 2023-01-17 05:04:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 25, 647218), True, datetime.datetime(2023, 1, 17, 5, 4, 25, 647218), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:35 UTC)" executed successfully builder stdout | 2023-01-17 05:04:26,095 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:26,108 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:26,108 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:26,118 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:04:27,444 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:04:27,476 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:04:28,346 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:04:28,887 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:04:28,890 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} exportactionlogsworker stdout | 2023-01-17 05:04:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:04:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:35.853396+00:00 (in 6.889216 seconds) exportactionlogsworker stdout | 2023-01-17 05:04:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:28 UTC)" (scheduled at 2023-01-17 05:04:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:04:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:04:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:28 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:04:29,277 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:04:29,380 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:04:30,662 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:04:30,664 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:04:30,667 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:04:30,669 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:04:30,671 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:04:32,445 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:04:32,799 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:04:33,902 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:04:33,995 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:04:34,121 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:04:34,665 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:04:34,668 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:04:34,671 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:04:34,685 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:04:34,909 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:04:34,912 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} manifestbackfillworker stdout | 2023-01-17 05:04:34,915 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-registry stdout | 2023-01-17 05:04:34,917 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:04:34,920 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:04:34,922 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:04:34,925 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:04:34,927 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:04:34,930 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:04:34,932 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} securityscanningnotificationworker stdout | 2023-01-17 05:04:35,150 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:04:35,328 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:04:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:45.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:04:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:45 UTC)" (scheduled at 2023-01-17 05:04:35.646354+00:00) notificationworker stdout | 2023-01-17 05:04:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 35, 647244), True, datetime.datetime(2023, 1, 17, 5, 4, 35, 647244), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:45 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:04:35,682 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:04:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:04:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:28.963662+00:00 (in 53.109752 seconds) exportactionlogsworker stdout | 2023-01-17 05:04:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:35 UTC)" (scheduled at 2023-01-17 05:04:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:04:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:04:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 35, 854276), True, datetime.datetime(2023, 1, 17, 5, 4, 35, 854276), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:04:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:04:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:04:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:35 UTC)" executed successfully builder stdout | 2023-01-17 05:04:36,128 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:36,140 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:36,141 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:36,151 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 05:04:36,207 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:04:36,311 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:04:36,475 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:04:37,060 [235] [DEBUG] [app] Starting request: urn:request:76835cfe-a5dc-4f06-bfba-9ed26869d24a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:37,060 [236] [DEBUG] [app] Starting request: urn:request:ee7d3008-c073-44fe-a564-4e3d7d47aacf (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:37,065 [242] [DEBUG] [app] Starting request: urn:request:9169a161-bfa1-4d39-b972-a4b4c946f90a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,065 [241] [DEBUG] [app] Starting request: urn:request:cc817561-475b-46a4-b501-425c2a143c35 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,066 [242] [DEBUG] [app] Ending request: urn:request:9169a161-bfa1-4d39-b972-a4b4c946f90a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9169a161-bfa1-4d39-b972-a4b4c946f90a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,066 [241] [DEBUG] [app] Ending request: urn:request:cc817561-475b-46a4-b501-425c2a143c35 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc817561-475b-46a4-b501-425c2a143c35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:04:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:04:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:37,070 [231] [DEBUG] [app] Starting request: urn:request:a24efe52-3760-4203-95c4-91a92e796070 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:37,070 [231] [DEBUG] [app] Ending request: urn:request:a24efe52-3760-4203-95c4-91a92e796070 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a24efe52-3760-4203-95c4-91a92e796070', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:37,070 [235] [DEBUG] [app] Starting request: urn:request:f041700d-f854-415d-b369-1800ff180e6c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:37,071 [235] [DEBUG] [app] Ending request: urn:request:f041700d-f854-415d-b369-1800ff180e6c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f041700d-f854-415d-b369-1800ff180e6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:04:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:37,087 [235] [DEBUG] [app] Ending request: urn:request:76835cfe-a5dc-4f06-bfba-9ed26869d24a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76835cfe-a5dc-4f06-bfba-9ed26869d24a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:37,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:37,087 [236] [DEBUG] [app] Ending request: urn:request:ee7d3008-c073-44fe-a564-4e3d7d47aacf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee7d3008-c073-44fe-a564-4e3d7d47aacf', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:04:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:04:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:04:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:37.537407+00:00 (in 0.280330 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:04:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:37 UTC)" (scheduled at 2023-01-17 05:04:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:04:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:04:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 37, 257511), True, datetime.datetime(2023, 1, 17, 5, 4, 37, 257511), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:04:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:04:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:04:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:04:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:04:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:37.256652+00:00 (in 59.718847 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:04:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:37 UTC)" (scheduled at 2023-01-17 05:04:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:04:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:04:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:04:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:04:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:10.738568+00:00 (in 29.999573 seconds) buildlogsarchiver stdout | 2023-01-17 05:04:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:10 UTC)" (scheduled at 2023-01-17 05:04:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:04:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 4, 40, 739363), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:04:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:04:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:04:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:04:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:04:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:42.941536+00:00 (in 1.792800 seconds) namespacegcworker stdout | 2023-01-17 05:04:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:41 UTC)" (scheduled at 2023-01-17 05:04:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:04:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:04:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 41, 149166), True, datetime.datetime(2023, 1, 17, 5, 4, 41, 149166), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:04:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:04:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:04:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:04:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:04:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:41.148245+00:00 (in 58.206219 seconds) namespacegcworker stdout | 2023-01-17 05:04:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:42 UTC)" (scheduled at 2023-01-17 05:04:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:04:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:04:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:05:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:04:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:55.646354+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 05:04:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:55 UTC)" (scheduled at 2023-01-17 05:04:45.646354+00:00) notificationworker stdout | 2023-01-17 05:04:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 45, 647244), True, datetime.datetime(2023, 1, 17, 5, 4, 45, 647244), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:04:55 UTC)" executed successfully builder stdout | 2023-01-17 05:04:46,153 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:46,167 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:46,167 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:46,178 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:04:49.537472+00:00 (in 2.600027 seconds) securityworker stdout | 2023-01-17 05:04:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:16 UTC)" (scheduled at 2023-01-17 05:04:46.936988+00:00) securityworker stdout | 2023-01-17 05:04:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:04:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:04:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:16.936988+00:00 (in 27.399061 seconds) securityworker stdout | 2023-01-17 05:04:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:19 UTC)" (scheduled at 2023-01-17 05:04:49.537472+00:00) securityworker stdout | 2023-01-17 05:04:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:04:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:04:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:04:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:04:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:04:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:04:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:04:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:04:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:04:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:04:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:04:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:04:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:04:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:04:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:04:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:21.348605+00:00 (in 29.999523 seconds) gcworker stdout | 2023-01-17 05:04:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:21 UTC)" (scheduled at 2023-01-17 05:04:51.348605+00:00) gcworker stdout | 2023-01-17 05:04:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722291349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:04:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:04:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:04:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:04:52,060 [233] [DEBUG] [app] Starting request: urn:request:e8e6cb5d-616f-449e-8b12-24cacab8bbaa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:52,061 [231] [DEBUG] [app] Starting request: urn:request:ef1cbaff-2dab-44f7-8d1a-8a96958f3faa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:04:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:52,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:04:52,065 [230] [DEBUG] [app] Starting request: urn:request:ba803eda-aa29-42ab-b4d9-049f4a410472 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:52,066 [243] [DEBUG] [app] Starting request: urn:request:e46077c1-9075-436a-a4e3-ddbad889fdab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:04:52,066 [230] [DEBUG] [app] Ending request: urn:request:ba803eda-aa29-42ab-b4d9-049f4a410472 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ba803eda-aa29-42ab-b4d9-049f4a410472', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:04:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:04:52,066 [243] [DEBUG] [app] Ending request: urn:request:e46077c1-9075-436a-a4e3-ddbad889fdab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e46077c1-9075-436a-a4e3-ddbad889fdab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:04:52,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:52,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:04:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:04:52,070 [236] [DEBUG] [app] Starting request: urn:request:2ade7427-995e-448f-88da-c38e54303558 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:52,071 [236] [DEBUG] [app] Ending request: urn:request:2ade7427-995e-448f-88da-c38e54303558 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2ade7427-995e-448f-88da-c38e54303558', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:04:52,071 [235] [DEBUG] [app] Starting request: urn:request:2f578fbd-d30e-40d5-aaa3-bdd38361a038 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:04:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:04:52,071 [235] [DEBUG] [app] Ending request: urn:request:2f578fbd-d30e-40d5-aaa3-bdd38361a038 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f578fbd-d30e-40d5-aaa3-bdd38361a038', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:04:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:04:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:52,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:04:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:04:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:04:52,087 [231] [DEBUG] [app] Ending request: urn:request:ef1cbaff-2dab-44f7-8d1a-8a96958f3faa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ef1cbaff-2dab-44f7-8d1a-8a96958f3faa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:52,087 [233] [DEBUG] [app] Ending request: urn:request:e8e6cb5d-616f-449e-8b12-24cacab8bbaa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e8e6cb5d-616f-449e-8b12-24cacab8bbaa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:04:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:04:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:04:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:04:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:04:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:01.636362+00:00 (in 5.989543 seconds) notificationworker stdout | 2023-01-17 05:04:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:05 UTC)" (scheduled at 2023-01-17 05:04:55.646354+00:00) notificationworker stdout | 2023-01-17 05:04:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:04:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 4, 55, 647216), True, datetime.datetime(2023, 1, 17, 5, 4, 55, 647216), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:04:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:04:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:04:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:05 UTC)" executed successfully builder stdout | 2023-01-17 05:04:56,188 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:04:56,200 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:04:56,200 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:04:56,211 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:04:57,480 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:04:57,511 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:04:58,371 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:04:58,903 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:04:58,922 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:04:59,286 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:04:59,415 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:05:00,671 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:05:00,673 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:05:00,676 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:05:00,678 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:05:00,680 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:05:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:05.646354+00:00 (in 4.009508 seconds) notificationworker stdout | 2023-01-17 05:05:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:01 UTC)" (scheduled at 2023-01-17 05:05:01.636362+00:00) notificationworker stdout | 2023-01-17 05:05:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:05:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:05:02,481 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:05:02,835 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:05:03,923 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:05:04,029 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:05:04,156 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:05:04,672 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:05:04,675 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:05:04,678 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:05:04,702 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:05:04,916 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:05:04,923 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:05:04,926 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:05:04,928 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:05:04,931 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:05:04,933 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:05:04,935 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} repositorygcworker stdout | 2023-01-17 05:05:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:05:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:19.346120+00:00 (in 14.408846 seconds) repositorygcworker stdout | 2023-01-17 05:05:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:04 UTC)" (scheduled at 2023-01-17 05:05:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:05:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. gunicorn-registry stdout | 2023-01-17 05:05:04,937 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:05:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 4, 937674), True, datetime.datetime(2023, 1, 17, 5, 5, 4, 937674), 0, 'repositorygc/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:05:04,940 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:05:04,948 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositorygcworker stdout | 2023-01-17 05:05:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:05:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:05:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:04 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:05:05,165 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:05:05,343 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:05:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:15.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 05:05:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:15 UTC)" (scheduled at 2023-01-17 05:05:05.646354+00:00) notificationworker stdout | 2023-01-17 05:05:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 5, 647220), True, datetime.datetime(2023, 1, 17, 5, 5, 5, 647220), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:05:05,717 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 05:05:06,220 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:06,232 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:06,232 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics teamsyncworker stdout | 2023-01-17 05:05:06,242 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} builder stdout | 2023-01-17 05:05:06,242 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:05:06,344 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:05:06,505 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:05:07,060 [235] [DEBUG] [app] Starting request: urn:request:d85a8671-cf46-4862-87e7-a8fe7551c56d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:07,060 [231] [DEBUG] [app] Starting request: urn:request:46b179e2-9ea4-4717-ac69-51ff6b729b16 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:07,064 [230] [DEBUG] [app] Starting request: urn:request:92f2e8cc-530d-4acb-9197-8a43d7713696 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:07,064 [241] [DEBUG] [app] Starting request: urn:request:b13000d7-0f84-4bea-bcfd-50190ed22359 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:07,065 [230] [DEBUG] [app] Ending request: urn:request:92f2e8cc-530d-4acb-9197-8a43d7713696 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:92f2e8cc-530d-4acb-9197-8a43d7713696', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:07,065 [241] [DEBUG] [app] Ending request: urn:request:b13000d7-0f84-4bea-bcfd-50190ed22359 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b13000d7-0f84-4bea-bcfd-50190ed22359', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:05:07,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:05:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:07,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:07,070 [236] [DEBUG] [app] Starting request: urn:request:742aa1a4-9788-4209-ae28-2ecb92abe797 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:07,070 [236] [DEBUG] [app] Ending request: urn:request:742aa1a4-9788-4209-ae28-2ecb92abe797 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:742aa1a4-9788-4209-ae28-2ecb92abe797', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:07,070 [233] [DEBUG] [app] Starting request: urn:request:af4934fd-abd8-4690-b660-fc8ca193332d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:07,071 [233] [DEBUG] [app] Ending request: urn:request:af4934fd-abd8-4690-b660-fc8ca193332d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af4934fd-abd8-4690-b660-fc8ca193332d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:05:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:07,087 [235] [DEBUG] [app] Ending request: urn:request:d85a8671-cf46-4862-87e7-a8fe7551c56d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d85a8671-cf46-4862-87e7-a8fe7551c56d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:07,087 [231] [DEBUG] [app] Ending request: urn:request:46b179e2-9ea4-4717-ac69-51ff6b729b16 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:46b179e2-9ea4-4717-ac69-51ff6b729b16', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:05:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) buildlogsarchiver stdout | 2023-01-17 05:05:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:05:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:40.738568+00:00 (in 29.999556 seconds) buildlogsarchiver stdout | 2023-01-17 05:05:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:40 UTC)" (scheduled at 2023-01-17 05:05:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:05:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 5, 10, 739377), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:05:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:05:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:05:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:05:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:25.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:05:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:25 UTC)" (scheduled at 2023-01-17 05:05:15.646354+00:00) notificationworker stdout | 2023-01-17 05:05:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 15, 647241), True, datetime.datetime(2023, 1, 17, 5, 5, 15, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:25 UTC)" executed successfully builder stdout | 2023-01-17 05:05:16,253 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:16,265 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:16,265 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:16,276 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:19.537472+00:00 (in 2.599982 seconds) securityworker stdout | 2023-01-17 05:05:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:46 UTC)" (scheduled at 2023-01-17 05:05:16.936988+00:00) securityworker stdout | 2023-01-17 05:05:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:05:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:05:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:04.936835+00:00 (in 45.590278 seconds) repositorygcworker stdout | 2023-01-17 05:05:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:19 UTC)" (scheduled at 2023-01-17 05:05:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:05:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:05:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:05:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:46.936988+00:00 (in 27.399032 seconds) securityworker stdout | 2023-01-17 05:05:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:49 UTC)" (scheduled at 2023-01-17 05:05:19.537472+00:00) securityworker stdout | 2023-01-17 05:05:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:05:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:05:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:05:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:05:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:05:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:05:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:05:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:05:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:51.348605+00:00 (in 29.999494 seconds) gcworker stdout | 2023-01-17 05:05:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:51 UTC)" (scheduled at 2023-01-17 05:05:21.348605+00:00) gcworker stdout | 2023-01-17 05:05:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722321349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:05:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:05:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:05:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:05:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:05:22,060 [231] [DEBUG] [app] Starting request: urn:request:75f2aa61-7727-46b3-b58a-7aa1dc2a0930 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:22,060 [236] [DEBUG] [app] Starting request: urn:request:a48e39fa-df41-45b2-ae0c-c683dcd4c8e1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:22,065 [230] [DEBUG] [app] Starting request: urn:request:6cb04c64-5222-458b-b85c-8bd401f3446f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,065 [243] [DEBUG] [app] Starting request: urn:request:18aedaa8-7bcd-417a-8521-b4dbc50841bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,065 [230] [DEBUG] [app] Ending request: urn:request:6cb04c64-5222-458b-b85c-8bd401f3446f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6cb04c64-5222-458b-b85c-8bd401f3446f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,066 [243] [DEBUG] [app] Ending request: urn:request:18aedaa8-7bcd-417a-8521-b4dbc50841bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18aedaa8-7bcd-417a-8521-b4dbc50841bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:05:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:05:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:05:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:22,071 [235] [DEBUG] [app] Starting request: urn:request:c202ead5-b16f-4d06-80e0-37ec3588df7e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:22,071 [235] [DEBUG] [app] Ending request: urn:request:c202ead5-b16f-4d06-80e0-37ec3588df7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c202ead5-b16f-4d06-80e0-37ec3588df7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:22,071 [236] [DEBUG] [app] Starting request: urn:request:065850d2-e5a3-402c-9506-be025ddd47e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:22,071 [236] [DEBUG] [app] Ending request: urn:request:065850d2-e5a3-402c-9506-be025ddd47e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:065850d2-e5a3-402c-9506-be025ddd47e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:05:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:22,088 [236] [DEBUG] [app] Ending request: urn:request:a48e39fa-df41-45b2-ae0c-c683dcd4c8e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a48e39fa-df41-45b2-ae0c-c683dcd4c8e1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:22,088 [231] [DEBUG] [app] Ending request: urn:request:75f2aa61-7727-46b3-b58a-7aa1dc2a0930 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75f2aa61-7727-46b3-b58a-7aa1dc2a0930', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:05:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:05:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:35.646354+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:05:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:35 UTC)" (scheduled at 2023-01-17 05:05:25.646354+00:00) notificationworker stdout | 2023-01-17 05:05:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 25, 647243), True, datetime.datetime(2023, 1, 17, 5, 5, 25, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:35 UTC)" executed successfully builder stdout | 2023-01-17 05:05:26,286 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:26,298 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:26,298 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:26,309 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:05:27,507 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:05:27,535 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:05:28,407 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:05:28,938 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:05:28,957 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} exportactionlogsworker stdout | 2023-01-17 05:05:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:05:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:35.853396+00:00 (in 6.889294 seconds) exportactionlogsworker stdout | 2023-01-17 05:05:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:28 UTC)" (scheduled at 2023-01-17 05:05:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:05:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:05:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:28 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:05:29,323 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:05:29,442 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:05:30,679 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:05:30,681 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:05:30,683 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:05:30,686 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:05:30,688 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:05:32,489 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:05:32,841 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:05:33,953 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:05:34,046 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:05:34,191 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:05:34,680 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:05:34,682 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:05:34,685 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:05:34,709 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:05:34,924 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:05:34,928 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:05:34,932 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:05:34,939 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:05:34,941 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:05:34,943 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:05:34,946 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:05:34,949 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:05:34,951 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:05:34,971 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:05:35,170 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:05:35,378 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:05:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:45.646354+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 05:05:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:45 UTC)" (scheduled at 2023-01-17 05:05:35.646354+00:00) notificationworker stdout | 2023-01-17 05:05:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 35, 647235), True, datetime.datetime(2023, 1, 17, 5, 5, 35, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:45 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:05:35,722 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:05:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:05:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:28.963662+00:00 (in 53.109790 seconds) exportactionlogsworker stdout | 2023-01-17 05:05:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:35 UTC)" (scheduled at 2023-01-17 05:05:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:05:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:05:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 35, 854252), True, datetime.datetime(2023, 1, 17, 5, 5, 35, 854252), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:05:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:05:35,864 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:05:35,864 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:05:36,270 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} builder stdout | 2023-01-17 05:05:36,314 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:36,326 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:36,326 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:36,336 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:05:36,370 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:05:36,538 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:05:37,060 [231] [DEBUG] [app] Starting request: urn:request:41bf11bc-b190-457e-a15e-17ed7dcc76f6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:37,060 [236] [DEBUG] [app] Starting request: urn:request:cd8e7bf0-bf36-4aa4-a6f6-8c5f9c4e0c3b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:37,065 [243] [DEBUG] [app] Starting request: urn:request:6322511c-ac16-4455-bf02-e0d1a16e640e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:37,065 [242] [DEBUG] [app] Starting request: urn:request:ec9ddc54-3de9-4840-8933-588073f605f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:37,065 [243] [DEBUG] [app] Ending request: urn:request:6322511c-ac16-4455-bf02-e0d1a16e640e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6322511c-ac16-4455-bf02-e0d1a16e640e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:05:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:05:37,066 [242] [DEBUG] [app] Ending request: urn:request:ec9ddc54-3de9-4840-8933-588073f605f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec9ddc54-3de9-4840-8933-588073f605f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:05:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:05:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:37,070 [235] [DEBUG] [app] Starting request: urn:request:8ddd0c86-340b-4873-803e-0bfee7d2454c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:37,070 [235] [DEBUG] [app] Ending request: urn:request:8ddd0c86-340b-4873-803e-0bfee7d2454c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ddd0c86-340b-4873-803e-0bfee7d2454c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:37,070 [233] [DEBUG] [app] Starting request: urn:request:9634f105-0164-44fb-8537-ceb477c55401 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:37,070 [233] [DEBUG] [app] Ending request: urn:request:9634f105-0164-44fb-8537-ceb477c55401 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9634f105-0164-44fb-8537-ceb477c55401', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:37,087 [236] [DEBUG] [app] Ending request: urn:request:cd8e7bf0-bf36-4aa4-a6f6-8c5f9c4e0c3b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd8e7bf0-bf36-4aa4-a6f6-8c5f9c4e0c3b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:37,088 [231] [DEBUG] [app] Ending request: urn:request:41bf11bc-b190-457e-a15e-17ed7dcc76f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:41bf11bc-b190-457e-a15e-17ed7dcc76f6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:05:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:05:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:05:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:37.537407+00:00 (in 0.280348 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:05:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:37 UTC)" (scheduled at 2023-01-17 05:05:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:05:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:05:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 37, 257496), True, datetime.datetime(2023, 1, 17, 5, 5, 37, 257496), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:05:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:05:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:05:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:05:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:05:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:23.537048+00:00 (in 45.999275 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:05:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:37 UTC)" (scheduled at 2023-01-17 05:05:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:05:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:05:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:05:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:05:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:10.738568+00:00 (in 29.999585 seconds) buildlogsarchiver stdout | 2023-01-17 05:05:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:10 UTC)" (scheduled at 2023-01-17 05:05:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:05:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 5, 40, 739357), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:05:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:05:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:05:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:05:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:05:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:42.941536+00:00 (in 1.792814 seconds) namespacegcworker stdout | 2023-01-17 05:05:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:41 UTC)" (scheduled at 2023-01-17 05:05:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:05:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:05:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 41, 149097), True, datetime.datetime(2023, 1, 17, 5, 5, 41, 149097), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:05:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:05:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:05:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:05:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:05:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:41.148245+00:00 (in 58.206266 seconds) namespacegcworker stdout | 2023-01-17 05:05:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:42 UTC)" (scheduled at 2023-01-17 05:05:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:05:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:05:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:06:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:05:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:55.646354+00:00 (in 9.999503 seconds) notificationworker stdout | 2023-01-17 05:05:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:55 UTC)" (scheduled at 2023-01-17 05:05:45.646354+00:00) notificationworker stdout | 2023-01-17 05:05:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 45, 647283), True, datetime.datetime(2023, 1, 17, 5, 5, 45, 647283), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:05:55 UTC)" executed successfully builder stdout | 2023-01-17 05:05:46,346 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:46,359 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:46,359 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:46,370 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:05:49.537472+00:00 (in 2.600026 seconds) securityworker stdout | 2023-01-17 05:05:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:16 UTC)" (scheduled at 2023-01-17 05:05:46.936988+00:00) securityworker stdout | 2023-01-17 05:05:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:05:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:05:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:16.936988+00:00 (in 27.399071 seconds) securityworker stdout | 2023-01-17 05:05:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:19 UTC)" (scheduled at 2023-01-17 05:05:49.537472+00:00) securityworker stdout | 2023-01-17 05:05:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:05:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:05:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:05:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:05:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:05:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:05:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:05:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:05:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:05:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:05:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:05:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:05:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:05:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:05:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:05:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:21.348605+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 05:05:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:21 UTC)" (scheduled at 2023-01-17 05:05:51.348605+00:00) gcworker stdout | 2023-01-17 05:05:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722351349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:05:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:05:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:05:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:05:52,060 [235] [DEBUG] [app] Starting request: urn:request:47b700c1-0beb-4f12-8090-d503d67abd75 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:52,060 [233] [DEBUG] [app] Starting request: urn:request:ee091ec6-c05f-4191-89a1-e9760ac7e870 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:05:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:05:52,065 [230] [DEBUG] [app] Starting request: urn:request:a814f7b6-15a8-4449-829e-c6b1b8f6aaaa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:52,065 [239] [DEBUG] [app] Starting request: urn:request:e355d963-d0a5-4d36-aa44-d49d084ef74e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:05:52,065 [230] [DEBUG] [app] Ending request: urn:request:a814f7b6-15a8-4449-829e-c6b1b8f6aaaa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a814f7b6-15a8-4449-829e-c6b1b8f6aaaa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:05:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:05:52,066 [239] [DEBUG] [app] Ending request: urn:request:e355d963-d0a5-4d36-aa44-d49d084ef74e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e355d963-d0a5-4d36-aa44-d49d084ef74e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:05:52,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:05:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:05:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:05:52,070 [231] [DEBUG] [app] Starting request: urn:request:9d268fd0-4daf-4082-95d0-aad8c0c771af (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:52,070 [236] [DEBUG] [app] Starting request: urn:request:c2d2847c-9280-4d8f-aaf4-4e472d4ea026 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:05:52,070 [231] [DEBUG] [app] Ending request: urn:request:9d268fd0-4daf-4082-95d0-aad8c0c771af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9d268fd0-4daf-4082-95d0-aad8c0c771af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:05:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:52,071 [236] [DEBUG] [app] Ending request: urn:request:c2d2847c-9280-4d8f-aaf4-4e472d4ea026 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2d2847c-9280-4d8f-aaf4-4e472d4ea026', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:05:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:05:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:05:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:05:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:05:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:05:52,088 [233] [DEBUG] [app] Ending request: urn:request:ee091ec6-c05f-4191-89a1-e9760ac7e870 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee091ec6-c05f-4191-89a1-e9760ac7e870', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:52,088 [235] [DEBUG] [app] Ending request: urn:request:47b700c1-0beb-4f12-8090-d503d67abd75 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47b700c1-0beb-4f12-8090-d503d67abd75', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:05:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:05:52,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:05:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:05:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:05:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:05:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:01.636362+00:00 (in 5.989530 seconds) notificationworker stdout | 2023-01-17 05:05:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:05 UTC)" (scheduled at 2023-01-17 05:05:55.646354+00:00) notificationworker stdout | 2023-01-17 05:05:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:05:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 5, 55, 647240), True, datetime.datetime(2023, 1, 17, 5, 5, 55, 647240), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:05:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:05:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:05:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:05 UTC)" executed successfully builder stdout | 2023-01-17 05:05:56,380 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:05:56,393 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:05:56,393 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:05:56,403 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:05:57,537 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:05:57,539 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:05:58,420 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:05:58,945 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:05:58,970 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:05:59,345 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:05:59,477 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:06:00,686 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:06:00,688 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:06:00,691 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:06:00,693 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:06:00,695 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:06:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:05.646354+00:00 (in 4.009541 seconds) notificationworker stdout | 2023-01-17 05:06:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:01 UTC)" (scheduled at 2023-01-17 05:06:01.636362+00:00) notificationworker stdout | 2023-01-17 05:06:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:06:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:06:02,525 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:06:02,870 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:06:03,983 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:06:04,081 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:06:04,196 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:06:04,687 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:06:04,690 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:06:04,692 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:06:04,739 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:06:04,930 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} repositorygcworker stdout | 2023-01-17 05:06:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:06:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:17.345601+00:00 (in 12.408342 seconds) repositorygcworker stdout | 2023-01-17 05:06:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:04 UTC)" (scheduled at 2023-01-17 05:06:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:06:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. gunicorn-registry stdout | 2023-01-17 05:06:04,938 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} repositorygcworker stdout | 2023-01-17 05:06:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 4, 937642), True, datetime.datetime(2023, 1, 17, 5, 6, 4, 937642), 0, 'repositorygc/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:06:04,940 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} repositorygcworker stdout | 2023-01-17 05:06:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:06:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:06:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:06:04,950 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:06:04,953 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:06:04,955 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:06:04,958 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:06:04,960 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:06:04,962 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:06:04,977 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:06:05,201 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:06:05,386 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:06:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:15.646354+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 05:06:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:15 UTC)" (scheduled at 2023-01-17 05:06:05.646354+00:00) notificationworker stdout | 2023-01-17 05:06:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 5, 647235), True, datetime.datetime(2023, 1, 17, 5, 6, 5, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:06:05,741 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:06:06,305 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:06:06,380 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:06:06,413 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:06,426 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:06,426 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:06,437 [60] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 05:06:06,543 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:06:07,060 [236] [DEBUG] [app] Starting request: urn:request:565dd255-c442-4063-ac42-c024cd6de83a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:07,060 [231] [DEBUG] [app] Starting request: urn:request:93b786c3-f18d-49cd-bb56-c042401c996b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:07,066 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:07,066 [242] [DEBUG] [app] Starting request: urn:request:dee07208-d5e3-4bf1-8fe7-13efe026af45 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:07,066 [243] [DEBUG] [app] Starting request: urn:request:91aa228a-27a5-404b-9d46-a7b0a83da992 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:07,066 [242] [DEBUG] [app] Ending request: urn:request:dee07208-d5e3-4bf1-8fe7-13efe026af45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dee07208-d5e3-4bf1-8fe7-13efe026af45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:06:07,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:06:07,067 [243] [DEBUG] [app] Ending request: urn:request:91aa228a-27a5-404b-9d46-a7b0a83da992 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:91aa228a-27a5-404b-9d46-a7b0a83da992', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:06:07,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:06:07,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:07,069 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,069 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:07,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:07,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:07,072 [235] [DEBUG] [app] Starting request: urn:request:515ea661-460d-469f-ab98-ee53d958e6b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:07,072 [233] [DEBUG] [app] Starting request: urn:request:74d66e28-e835-421e-a68e-6bdf84998d99 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:07,072 [235] [DEBUG] [app] Ending request: urn:request:515ea661-460d-469f-ab98-ee53d958e6b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:515ea661-460d-469f-ab98-ee53d958e6b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:06:07,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:07,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:07,073 [233] [DEBUG] [app] Ending request: urn:request:74d66e28-e835-421e-a68e-6bdf84998d99 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:74d66e28-e835-421e-a68e-6bdf84998d99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:07,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:06:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:07,073 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:07,074 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:07,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:07,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:07,089 [236] [DEBUG] [app] Ending request: urn:request:565dd255-c442-4063-ac42-c024cd6de83a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:565dd255-c442-4063-ac42-c024cd6de83a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:07,089 [231] [DEBUG] [app] Ending request: urn:request:93b786c3-f18d-49cd-bb56-c042401c996b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93b786c3-f18d-49cd-bb56-c042401c996b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:07,090 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:07,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:07,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:06:07,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) buildlogsarchiver stdout | 2023-01-17 05:06:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:06:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:40.738568+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 05:06:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:40 UTC)" (scheduled at 2023-01-17 05:06:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:06:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 6, 10, 739422), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:06:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:06:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:06:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:06:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:25.646354+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 05:06:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:25 UTC)" (scheduled at 2023-01-17 05:06:15.646354+00:00) notificationworker stdout | 2023-01-17 05:06:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 15, 647227), True, datetime.datetime(2023, 1, 17, 5, 6, 15, 647227), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:25 UTC)" executed successfully builder stdout | 2023-01-17 05:06:16,437 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:16,450 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:16,450 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:16,460 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:19.537472+00:00 (in 2.600041 seconds) securityworker stdout | 2023-01-17 05:06:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:46 UTC)" (scheduled at 2023-01-17 05:06:16.936988+00:00) securityworker stdout | 2023-01-17 05:06:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:06:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:06:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:19.346120+00:00 (in 2.000046 seconds) repositorygcworker stdout | 2023-01-17 05:06:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:17 UTC)" (scheduled at 2023-01-17 05:06:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:06:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 6, 17, 346467), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:06:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 17, 346467), True, datetime.datetime(2023, 1, 17, 5, 6, 17, 346467), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:06:17,360 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 17, 346467), True, datetime.datetime(2023, 1, 17, 5, 6, 17, 346467), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 6, 17, 346467), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:06:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:06:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:06:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:06:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:04.936835+00:00 (in 45.590274 seconds) repositorygcworker stdout | 2023-01-17 05:06:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:19 UTC)" (scheduled at 2023-01-17 05:06:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:06:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:06:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:06:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:46.936988+00:00 (in 27.399059 seconds) securityworker stdout | 2023-01-17 05:06:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:49 UTC)" (scheduled at 2023-01-17 05:06:19.537472+00:00) securityworker stdout | 2023-01-17 05:06:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:06:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:06:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:06:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:06:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:06:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:06:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:06:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:06:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:51.348605+00:00 (in 29.999515 seconds) gcworker stdout | 2023-01-17 05:06:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:51 UTC)" (scheduled at 2023-01-17 05:06:21.348605+00:00) gcworker stdout | 2023-01-17 05:06:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722381349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:06:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:06:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:06:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:06:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:06:22,060 [231] [DEBUG] [app] Starting request: urn:request:186a3a55-6db1-4e4c-8d31-d5eb98d589eb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:22,060 [236] [DEBUG] [app] Starting request: urn:request:679b3d52-6b12-4943-a6d6-e3206a65b47c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:22,065 [230] [DEBUG] [app] Starting request: urn:request:34c33147-c7b6-41b9-9640-44845ff71600 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:22,065 [243] [DEBUG] [app] Starting request: urn:request:3d640eeb-37b8-458f-aec6-5a6ffc3a8c23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:22,066 [230] [DEBUG] [app] Ending request: urn:request:34c33147-c7b6-41b9-9640-44845ff71600 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34c33147-c7b6-41b9-9640-44845ff71600', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:06:22,066 [243] [DEBUG] [app] Ending request: urn:request:3d640eeb-37b8-458f-aec6-5a6ffc3a8c23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d640eeb-37b8-458f-aec6-5a6ffc3a8c23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:06:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:06:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:22,071 [235] [DEBUG] [app] Starting request: urn:request:db956f70-0ded-43d6-a549-db685fb3d967 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:22,071 [231] [DEBUG] [app] Starting request: urn:request:bc647e61-0c11-42ad-8751-44989e7689cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:22,071 [235] [DEBUG] [app] Ending request: urn:request:db956f70-0ded-43d6-a549-db685fb3d967 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db956f70-0ded-43d6-a549-db685fb3d967', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:22,071 [231] [DEBUG] [app] Ending request: urn:request:bc647e61-0c11-42ad-8751-44989e7689cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc647e61-0c11-42ad-8751-44989e7689cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:06:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:22,088 [231] [DEBUG] [app] Ending request: urn:request:186a3a55-6db1-4e4c-8d31-d5eb98d589eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:186a3a55-6db1-4e4c-8d31-d5eb98d589eb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:22,088 [236] [DEBUG] [app] Ending request: urn:request:679b3d52-6b12-4943-a6d6-e3206a65b47c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:679b3d52-6b12-4943-a6d6-e3206a65b47c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:06:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:06:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:06:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:37.256652+00:00 (in 13.719186 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:06:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:23 UTC)" (scheduled at 2023-01-17 05:06:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:06:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 6, 23, 537855), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:06:23,549 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 23, 537855), True, datetime.datetime(2023, 1, 17, 5, 6, 23, 537855), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:06:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 23, 537855), True, datetime.datetime(2023, 1, 17, 5, 6, 23, 537855), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 6, 23, 537855), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:06:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:06:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:06:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:35.646354+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 05:06:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:35 UTC)" (scheduled at 2023-01-17 05:06:25.646354+00:00) notificationworker stdout | 2023-01-17 05:06:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 25, 647224), True, datetime.datetime(2023, 1, 17, 5, 6, 25, 647224), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:35 UTC)" executed successfully builder stdout | 2023-01-17 05:06:26,471 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:26,484 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:26,484 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:26,494 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:06:27,558 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:06:27,574 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:06:28,450 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:06:28,962 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} exportactionlogsworker stdout | 2023-01-17 05:06:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:06:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:35.853396+00:00 (in 6.889302 seconds) exportactionlogsworker stdout | 2023-01-17 05:06:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:28 UTC)" (scheduled at 2023-01-17 05:06:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:06:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:06:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:28 UTC)" executed successfully logrotateworker stdout | 2023-01-17 05:06:28,986 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:06:29,370 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:06:29,513 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:06:30,693 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:06:30,695 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:06:30,698 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:06:30,701 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:06:30,703 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:06:32,535 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:06:32,891 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:06:34,007 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:06:34,090 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:06:34,206 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:06:34,695 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:06:34,697 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:06:34,699 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:06:34,757 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:06:34,942 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:06:34,944 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:06:34,949 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:06:34,959 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:06:34,962 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:06:34,964 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:06:34,967 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:06:34,970 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:06:34,972 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:06:35,012 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:06:35,229 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:06:35,398 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:06:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:45.646354+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:06:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:45 UTC)" (scheduled at 2023-01-17 05:06:35.646354+00:00) notificationworker stdout | 2023-01-17 05:06:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 35, 647245), True, datetime.datetime(2023, 1, 17, 5, 6, 35, 647245), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:45 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:06:35,758 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:06:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:06:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:54.963413+00:00 (in 19.109484 seconds) exportactionlogsworker stdout | 2023-01-17 05:06:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:35 UTC)" (scheduled at 2023-01-17 05:06:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:06:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:06:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 35, 854291), True, datetime.datetime(2023, 1, 17, 5, 6, 35, 854291), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:06:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:06:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:06:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:06:36,312 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:06:36,414 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 05:06:36,504 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:36,517 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:36,517 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:36,528 [60] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 05:06:36,554 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:06:37,060 [231] [DEBUG] [app] Starting request: urn:request:ecad255e-f0e9-42ac-866c-5d18b5adc6ec (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:37,060 [236] [DEBUG] [app] Starting request: urn:request:41d92ebe-beb1-4a0b-ab14-b8d7050b508e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:37,065 [230] [DEBUG] [app] Starting request: urn:request:58968442-7050-470d-a77e-b5a54b9f303f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:37,065 [243] [DEBUG] [app] Starting request: urn:request:4dcff03e-9183-4b76-941b-ad3b60cd5916 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:37,066 [230] [DEBUG] [app] Ending request: urn:request:58968442-7050-470d-a77e-b5a54b9f303f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58968442-7050-470d-a77e-b5a54b9f303f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:37,066 [243] [DEBUG] [app] Ending request: urn:request:4dcff03e-9183-4b76-941b-ad3b60cd5916 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4dcff03e-9183-4b76-941b-ad3b60cd5916', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:06:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:06:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:37,070 [231] [DEBUG] [app] Starting request: urn:request:780bcec0-4d5e-4439-913a-0bc0436a944f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:37,070 [236] [DEBUG] [app] Starting request: urn:request:4d4d947f-8067-4ec5-83e8-528242cbcb64 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:37,071 [236] [DEBUG] [app] Ending request: urn:request:4d4d947f-8067-4ec5-83e8-528242cbcb64 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d4d947f-8067-4ec5-83e8-528242cbcb64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:37,071 [231] [DEBUG] [app] Ending request: urn:request:780bcec0-4d5e-4439-913a-0bc0436a944f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:780bcec0-4d5e-4439-913a-0bc0436a944f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:06:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:37,088 [231] [DEBUG] [app] Ending request: urn:request:ecad255e-f0e9-42ac-866c-5d18b5adc6ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ecad255e-f0e9-42ac-866c-5d18b5adc6ec', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:06:37,092 [236] [DEBUG] [app] Ending request: urn:request:41d92ebe-beb1-4a0b-ab14-b8d7050b508e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:41d92ebe-beb1-4a0b-ab14-b8d7050b508e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:37,092 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:37,092 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.033 47 0.033) securityscanningnotificationworker stdout | 2023-01-17 05:06:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:06:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:37.537407+00:00 (in 0.280267 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:06:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:37 UTC)" (scheduled at 2023-01-17 05:06:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:06:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:06:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 37, 257572), True, datetime.datetime(2023, 1, 17, 5, 6, 37, 257572), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:06:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:06:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:06:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:06:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:06:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:37.256652+00:00 (in 59.718812 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:06:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:37 UTC)" (scheduled at 2023-01-17 05:06:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:06:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:06:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:06:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:06:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:10.738568+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 05:06:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:10 UTC)" (scheduled at 2023-01-17 05:06:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:06:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 6, 40, 739462), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:06:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:06:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:06:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:06:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:06:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:42.941536+00:00 (in 1.792840 seconds) namespacegcworker stdout | 2023-01-17 05:06:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:41 UTC)" (scheduled at 2023-01-17 05:06:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:06:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:06:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 41, 149125), True, datetime.datetime(2023, 1, 17, 5, 6, 41, 149125), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:06:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:06:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:06:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:06:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:06:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:06.941185+00:00 (in 23.999134 seconds) namespacegcworker stdout | 2023-01-17 05:06:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:42 UTC)" (scheduled at 2023-01-17 05:06:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:06:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:06:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:07:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:06:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:55.646354+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:06:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:55 UTC)" (scheduled at 2023-01-17 05:06:45.646354+00:00) notificationworker stdout | 2023-01-17 05:06:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 45, 647263), True, datetime.datetime(2023, 1, 17, 5, 6, 45, 647263), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:06:55 UTC)" executed successfully builder stdout | 2023-01-17 05:06:46,532 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:46,545 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:46,545 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:46,555 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:06:49.537472+00:00 (in 2.600054 seconds) securityworker stdout | 2023-01-17 05:06:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:16 UTC)" (scheduled at 2023-01-17 05:06:46.936988+00:00) securityworker stdout | 2023-01-17 05:06:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:06:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:06:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:16.936988+00:00 (in 27.399083 seconds) securityworker stdout | 2023-01-17 05:06:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:19 UTC)" (scheduled at 2023-01-17 05:06:49.537472+00:00) securityworker stdout | 2023-01-17 05:06:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:06:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:06:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:06:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:06:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:06:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:06:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:06:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:06:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:06:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:06:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:06:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:06:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:06:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:06:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:06:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:21.348605+00:00 (in 29.999556 seconds) gcworker stdout | 2023-01-17 05:06:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:21 UTC)" (scheduled at 2023-01-17 05:06:51.348605+00:00) gcworker stdout | 2023-01-17 05:06:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722411349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:06:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:06:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:06:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:06:52,060 [231] [DEBUG] [app] Starting request: urn:request:c2ac0639-a0c7-4df3-bf5f-4117331b2c74 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:52,060 [236] [DEBUG] [app] Starting request: urn:request:be0f3e86-928e-4f6c-9769-2d9091bb1578 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:06:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:06:52,065 [230] [DEBUG] [app] Starting request: urn:request:f41e2256-20a8-4c7c-9783-0a29538afbc3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:52,065 [230] [DEBUG] [app] Ending request: urn:request:f41e2256-20a8-4c7c-9783-0a29538afbc3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f41e2256-20a8-4c7c-9783-0a29538afbc3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:52,065 [243] [DEBUG] [app] Starting request: urn:request:a09e7829-549a-452f-ae10-d6be2f9ea950 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:06:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:06:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:06:52,066 [243] [DEBUG] [app] Ending request: urn:request:a09e7829-549a-452f-ae10-d6be2f9ea950 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a09e7829-549a-452f-ae10-d6be2f9ea950', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:06:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:06:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:06:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:52,069 [235] [DEBUG] [app] Starting request: urn:request:5c82f385-3225-49c4-b01d-47daae0a8070 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:52,070 [235] [DEBUG] [app] Ending request: urn:request:5c82f385-3225-49c4-b01d-47daae0a8070 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5c82f385-3225-49c4-b01d-47daae0a8070', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:06:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:06:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:06:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:52,071 [235] [DEBUG] [app] Starting request: urn:request:eec86d8e-88fe-4030-8c8a-ab7370783982 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:06:52,071 [235] [DEBUG] [app] Ending request: urn:request:eec86d8e-88fe-4030-8c8a-ab7370783982 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eec86d8e-88fe-4030-8c8a-ab7370783982', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:06:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.000 159 0.001) gunicorn-web stdout | 2023-01-17 05:06:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:06:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:06:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:06:52,087 [231] [DEBUG] [app] Ending request: urn:request:c2ac0639-a0c7-4df3-bf5f-4117331b2c74 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2ac0639-a0c7-4df3-bf5f-4117331b2c74', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:06:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:06:52,088 [236] [DEBUG] [app] Ending request: urn:request:be0f3e86-928e-4f6c-9769-2d9091bb1578 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:be0f3e86-928e-4f6c-9769-2d9091bb1578', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:06:52,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:06:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 05:06:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:06:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:28.963662+00:00 (in 33.999799 seconds) exportactionlogsworker stdout | 2023-01-17 05:06:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:54 UTC)" (scheduled at 2023-01-17 05:06:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:06:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 6, 54, 964225), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:06:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 54, 964225), True, datetime.datetime(2023, 1, 17, 5, 6, 54, 964225), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:06:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 6, 54, 964225), True, datetime.datetime(2023, 1, 17, 5, 6, 54, 964225), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 6, 54, 964225), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:06:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:06:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:11:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:06:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:06:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:01.636362+00:00 (in 5.989515 seconds) notificationworker stdout | 2023-01-17 05:06:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:05 UTC)" (scheduled at 2023-01-17 05:06:55.646354+00:00) notificationworker stdout | 2023-01-17 05:06:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:06:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 6, 55, 647247), True, datetime.datetime(2023, 1, 17, 5, 6, 55, 647247), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:06:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:06:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:06:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:05 UTC)" executed successfully builder stdout | 2023-01-17 05:06:56,560 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:06:56,573 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:06:56,573 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:06:56,584 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:06:57,588 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:06:57,603 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:06:58,462 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:06:58,969 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:06:59,021 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:06:59,388 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:06:59,550 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:07:00,701 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:07:00,704 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:07:00,706 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:07:00,708 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:07:00,710 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:07:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:05.646354+00:00 (in 4.009494 seconds) notificationworker stdout | 2023-01-17 05:07:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:01 UTC)" (scheduled at 2023-01-17 05:07:01.636362+00:00) notificationworker stdout | 2023-01-17 05:07:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:07:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:07:02,570 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:07:02,900 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:07:04,042 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:07:04,112 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:07:04,239 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:07:04,701 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:07:04,704 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:07:04,706 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:07:04,790 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} repositorygcworker stdout | 2023-01-17 05:07:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:07:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:19.346120+00:00 (in 14.408849 seconds) repositorygcworker stdout | 2023-01-17 05:07:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:04 UTC)" (scheduled at 2023-01-17 05:07:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:07:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:07:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 4, 937677), True, datetime.datetime(2023, 1, 17, 5, 7, 4, 937677), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:07:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:07:04,948 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:07:04,948 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:07:04,952 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:07:04,954 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:07:04,957 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:07:04,967 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:07:04,971 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:07:04,973 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:07:04,976 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:07:04,979 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:07:04,981 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:07:05,017 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:07:05,234 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:07:05,433 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:07:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:15.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:07:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:15 UTC)" (scheduled at 2023-01-17 05:07:05.646354+00:00) notificationworker stdout | 2023-01-17 05:07:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 5, 647187), True, datetime.datetime(2023, 1, 17, 5, 7, 5, 647187), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:07:05,767 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:07:06,326 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:07:06,429 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:07:06,559 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:07:06,586 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:06,598 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:06,598 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:06,609 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:07:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:07:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:41.148245+00:00 (in 34.206602 seconds) namespacegcworker stdout | 2023-01-17 05:07:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:12:06 UTC)" (scheduled at 2023-01-17 05:07:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:07:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 7, 6, 942023), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:07:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 7, 6, 942023), True, datetime.datetime(2023, 1, 17, 5, 7, 6, 942023), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:07:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 7, 6, 942023), True, datetime.datetime(2023, 1, 17, 5, 7, 6, 942023), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 7, 6, 942023), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:07:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:07:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:12:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:07:07,060 [231] [DEBUG] [app] Starting request: urn:request:e6f5a276-f6f5-43be-9e1a-8e19d90d42c6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:07,060 [236] [DEBUG] [app] Starting request: urn:request:63fb58e6-e872-455f-b8d0-91d473a7aa3e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:07,065 [242] [DEBUG] [app] Starting request: urn:request:830319c3-14fd-4514-adeb-adfacaa878a7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:07,066 [242] [DEBUG] [app] Ending request: urn:request:830319c3-14fd-4514-adeb-adfacaa878a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:830319c3-14fd-4514-adeb-adfacaa878a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:07,066 [230] [DEBUG] [app] Starting request: urn:request:1e3bf04f-642e-4c54-8a80-b860816e92cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:07:07,066 [230] [DEBUG] [app] Ending request: urn:request:1e3bf04f-642e-4c54-8a80-b860816e92cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e3bf04f-642e-4c54-8a80-b860816e92cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:07:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:07:07,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:07,071 [235] [DEBUG] [app] Starting request: urn:request:0fb67f67-1ce3-41e5-8d7a-7c56fc0e3f27 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:07,071 [235] [DEBUG] [app] Ending request: urn:request:0fb67f67-1ce3-41e5-8d7a-7c56fc0e3f27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0fb67f67-1ce3-41e5-8d7a-7c56fc0e3f27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:07,071 [233] [DEBUG] [app] Starting request: urn:request:5f4f30e0-cf94-4c0e-b7c6-0bb52b570c99 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:07,072 [233] [DEBUG] [app] Ending request: urn:request:5f4f30e0-cf94-4c0e-b7c6-0bb52b570c99 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f4f30e0-cf94-4c0e-b7c6-0bb52b570c99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:07:07,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:07,088 [231] [DEBUG] [app] Ending request: urn:request:e6f5a276-f6f5-43be-9e1a-8e19d90d42c6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6f5a276-f6f5-43be-9e1a-8e19d90d42c6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:07,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:07:07,088 [236] [DEBUG] [app] Ending request: urn:request:63fb58e6-e872-455f-b8d0-91d473a7aa3e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63fb58e6-e872-455f-b8d0-91d473a7aa3e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:07:07,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:07:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 05:07:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:07:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:40.738568+00:00 (in 29.999510 seconds) buildlogsarchiver stdout | 2023-01-17 05:07:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:40 UTC)" (scheduled at 2023-01-17 05:07:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:07:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 7, 10, 739468), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:07:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:07:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:07:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:07:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:25.646354+00:00 (in 9.999537 seconds) notificationworker stdout | 2023-01-17 05:07:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:25 UTC)" (scheduled at 2023-01-17 05:07:15.646354+00:00) notificationworker stdout | 2023-01-17 05:07:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 15, 647245), True, datetime.datetime(2023, 1, 17, 5, 7, 15, 647245), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:25 UTC)" executed successfully builder stdout | 2023-01-17 05:07:16,610 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:16,623 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:16,623 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:16,633 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:19.537472+00:00 (in 2.600044 seconds) securityworker stdout | 2023-01-17 05:07:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:46 UTC)" (scheduled at 2023-01-17 05:07:16.936988+00:00) securityworker stdout | 2023-01-17 05:07:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:07:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:07:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:04.936835+00:00 (in 45.590262 seconds) repositorygcworker stdout | 2023-01-17 05:07:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:19 UTC)" (scheduled at 2023-01-17 05:07:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:07:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:07:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:07:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:46.936988+00:00 (in 27.399054 seconds) securityworker stdout | 2023-01-17 05:07:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:49 UTC)" (scheduled at 2023-01-17 05:07:19.537472+00:00) securityworker stdout | 2023-01-17 05:07:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:07:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:07:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:07:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:07:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:07:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:07:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:07:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:07:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:51.348605+00:00 (in 29.999532 seconds) gcworker stdout | 2023-01-17 05:07:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:51 UTC)" (scheduled at 2023-01-17 05:07:21.348605+00:00) gcworker stdout | 2023-01-17 05:07:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722441349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:07:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:07:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:07:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:07:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:07:22,060 [231] [DEBUG] [app] Starting request: urn:request:d5b8b7ba-f22c-4e6d-9d31-1d9f198796d8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:22,060 [236] [DEBUG] [app] Starting request: urn:request:57fd9b8b-270d-4604-b2ce-014e98a3d1b1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:22,065 [230] [DEBUG] [app] Starting request: urn:request:a71d3f47-87cf-41c5-bcd8-7b02a99b4f3a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:22,065 [243] [DEBUG] [app] Starting request: urn:request:07ac08ae-673a-4362-9c81-f36f593d9d5c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:22,065 [230] [DEBUG] [app] Ending request: urn:request:a71d3f47-87cf-41c5-bcd8-7b02a99b4f3a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a71d3f47-87cf-41c5-bcd8-7b02a99b4f3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:07:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:07:22,065 [243] [DEBUG] [app] Ending request: urn:request:07ac08ae-673a-4362-9c81-f36f593d9d5c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07ac08ae-673a-4362-9c81-f36f593d9d5c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:07:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:07:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:22,069 [231] [DEBUG] [app] Starting request: urn:request:2195c3d9-5703-47ea-b31b-367681f60630 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:22,070 [231] [DEBUG] [app] Ending request: urn:request:2195c3d9-5703-47ea-b31b-367681f60630 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2195c3d9-5703-47ea-b31b-367681f60630', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:07:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:22,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:22,071 [235] [DEBUG] [app] Starting request: urn:request:34d3c564-f047-423b-8bdd-4b89ad693177 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:22,072 [235] [DEBUG] [app] Ending request: urn:request:34d3c564-f047-423b-8bdd-4b89ad693177 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:34d3c564-f047-423b-8bdd-4b89ad693177', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:07:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:22,087 [236] [DEBUG] [app] Ending request: urn:request:57fd9b8b-270d-4604-b2ce-014e98a3d1b1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:57fd9b8b-270d-4604-b2ce-014e98a3d1b1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:22,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:07:22,088 [231] [DEBUG] [app] Ending request: urn:request:d5b8b7ba-f22c-4e6d-9d31-1d9f198796d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d5b8b7ba-f22c-4e6d-9d31-1d9f198796d8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 05:07:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:35.646354+00:00 (in 9.999504 seconds) notificationworker stdout | 2023-01-17 05:07:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:35 UTC)" (scheduled at 2023-01-17 05:07:25.646354+00:00) notificationworker stdout | 2023-01-17 05:07:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 25, 647267), True, datetime.datetime(2023, 1, 17, 5, 7, 25, 647267), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:35 UTC)" executed successfully builder stdout | 2023-01-17 05:07:26,644 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:26,656 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:26,656 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:26,667 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:07:27,603 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:07:27,638 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:07:28,498 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:07:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:07:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:35.853396+00:00 (in 6.889237 seconds) exportactionlogsworker stdout | 2023-01-17 05:07:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:28 UTC)" (scheduled at 2023-01-17 05:07:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:07:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:07:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:28 UTC)" executed successfully builder stdout | 2023-01-17 05:07:28,978 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:07:29,037 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:07:29,403 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:07:29,569 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:07:30,709 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:07:30,711 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:07:30,714 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:07:30,716 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:07:30,719 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:07:32,577 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:07:32,935 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:07:34,070 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:07:34,143 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:07:34,274 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:07:34,709 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:07:34,712 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:07:34,714 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:07:34,825 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:07:34,962 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:07:34,965 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:07:34,967 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:07:34,974 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:07:34,979 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:07:34,982 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:07:34,985 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:07:34,987 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:07:34,989 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:07:35,048 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:07:35,269 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:07:35,445 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:07:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:45.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:07:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:45 UTC)" (scheduled at 2023-01-17 05:07:35.646354+00:00) notificationworker stdout | 2023-01-17 05:07:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 35, 647190), True, datetime.datetime(2023, 1, 17, 5, 7, 35, 647190), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:45 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:07:35,779 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:07:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:07:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:28.963662+00:00 (in 53.109776 seconds) exportactionlogsworker stdout | 2023-01-17 05:07:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:35 UTC)" (scheduled at 2023-01-17 05:07:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:07:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:07:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 35, 854258), True, datetime.datetime(2023, 1, 17, 5, 7, 35, 854258), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:07:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:07:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:07:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:07:36,337 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:07:36,455 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:07:36,594 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:07:36,677 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:36,690 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:36,690 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:36,701 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:37,060 [235] [DEBUG] [app] Starting request: urn:request:0e23b57c-700a-4be3-a3d9-760e33b0aabc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:37,060 [236] [DEBUG] [app] Starting request: urn:request:4f51bcc8-252f-478a-a670-c50c43243f96 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:37,065 [243] [DEBUG] [app] Starting request: urn:request:ea5bf15e-8812-46b5-b30f-fd39a5a9679d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:37,066 [242] [DEBUG] [app] Starting request: urn:request:0ae91f63-6141-4481-91eb-68f423036a00 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:07:37,066 [243] [DEBUG] [app] Ending request: urn:request:ea5bf15e-8812-46b5-b30f-fd39a5a9679d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea5bf15e-8812-46b5-b30f-fd39a5a9679d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:07:37,066 [242] [DEBUG] [app] Ending request: urn:request:0ae91f63-6141-4481-91eb-68f423036a00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ae91f63-6141-4481-91eb-68f423036a00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:07:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:07:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:07:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:37,070 [231] [DEBUG] [app] Starting request: urn:request:efb222df-801d-4252-944a-c6c7533c1512 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:37,071 [235] [DEBUG] [app] Starting request: urn:request:8ae4b489-a0ef-4c08-b0c6-f44d585fd391 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:37,071 [231] [DEBUG] [app] Ending request: urn:request:efb222df-801d-4252-944a-c6c7533c1512 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:efb222df-801d-4252-944a-c6c7533c1512', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:07:37,071 [235] [DEBUG] [app] Ending request: urn:request:8ae4b489-a0ef-4c08-b0c6-f44d585fd391 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ae4b489-a0ef-4c08-b0c6-f44d585fd391', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:07:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:37,088 [236] [DEBUG] [app] Ending request: urn:request:4f51bcc8-252f-478a-a670-c50c43243f96 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f51bcc8-252f-478a-a670-c50c43243f96', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:07:37,094 [235] [DEBUG] [app] Ending request: urn:request:0e23b57c-700a-4be3-a3d9-760e33b0aabc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e23b57c-700a-4be3-a3d9-760e33b0aabc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:37,094 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:37,094 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) securityscanningnotificationworker stdout | 2023-01-17 05:07:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:07:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:37.537407+00:00 (in 0.280346 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:07:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:37 UTC)" (scheduled at 2023-01-17 05:07:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:07:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:07:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 37, 257493), True, datetime.datetime(2023, 1, 17, 5, 7, 37, 257493), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:07:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:07:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:07:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:07:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:07:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:37.256652+00:00 (in 59.718853 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:07:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:37 UTC)" (scheduled at 2023-01-17 05:07:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:07:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:07:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:07:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:07:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:10.738568+00:00 (in 29.999497 seconds) buildlogsarchiver stdout | 2023-01-17 05:07:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:10 UTC)" (scheduled at 2023-01-17 05:07:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:07:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 7, 40, 739442), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:07:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:07:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:07:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:07:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:07:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:42.941536+00:00 (in 1.792833 seconds) namespacegcworker stdout | 2023-01-17 05:07:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:41 UTC)" (scheduled at 2023-01-17 05:07:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:07:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:07:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 41, 149125), True, datetime.datetime(2023, 1, 17, 5, 7, 41, 149125), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:07:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:07:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:07:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:07:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:07:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:41.148245+00:00 (in 58.206227 seconds) namespacegcworker stdout | 2023-01-17 05:07:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:42 UTC)" (scheduled at 2023-01-17 05:07:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:07:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:07:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:08:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:07:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:55.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:07:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:55 UTC)" (scheduled at 2023-01-17 05:07:45.646354+00:00) notificationworker stdout | 2023-01-17 05:07:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 45, 647223), True, datetime.datetime(2023, 1, 17, 5, 7, 45, 647223), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:07:55 UTC)" executed successfully builder stdout | 2023-01-17 05:07:46,711 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:46,724 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:46,724 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:46,735 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:07:49.537472+00:00 (in 2.600038 seconds) securityworker stdout | 2023-01-17 05:07:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:16 UTC)" (scheduled at 2023-01-17 05:07:46.936988+00:00) securityworker stdout | 2023-01-17 05:07:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:07:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:07:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:16.936988+00:00 (in 27.399090 seconds) securityworker stdout | 2023-01-17 05:07:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:19 UTC)" (scheduled at 2023-01-17 05:07:49.537472+00:00) securityworker stdout | 2023-01-17 05:07:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:07:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:07:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:49,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:07:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:07:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:07:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:07:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:07:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:07:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:07:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:07:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:07:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:07:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:07:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:07:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:07:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:21.348605+00:00 (in 29.999494 seconds) gcworker stdout | 2023-01-17 05:07:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:21 UTC)" (scheduled at 2023-01-17 05:07:51.348605+00:00) gcworker stdout | 2023-01-17 05:07:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722471349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:07:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:07:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:07:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:07:52,060 [236] [DEBUG] [app] Starting request: urn:request:f3765cfb-0beb-499e-93de-83c170c824f5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:52,060 [231] [DEBUG] [app] Starting request: urn:request:9c50c7a8-7773-4297-843e-f0924b258b69 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:07:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:52,065 [241] [DEBUG] [app] Starting request: urn:request:1fbcee6d-660a-4890-82ef-7908bf26f444 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:07:52,065 [241] [DEBUG] [app] Ending request: urn:request:1fbcee6d-660a-4890-82ef-7908bf26f444 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1fbcee6d-660a-4890-82ef-7908bf26f444', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:52,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:07:52,065 [230] [DEBUG] [app] Starting request: urn:request:531e7f63-67d0-4df8-948f-e7ecb23294df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:07:52,066 [230] [DEBUG] [app] Ending request: urn:request:531e7f63-67d0-4df8-948f-e7ecb23294df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:531e7f63-67d0-4df8-948f-e7ecb23294df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:07:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:07:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:07:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:52,070 [235] [DEBUG] [app] Starting request: urn:request:d4779f52-ab6f-459c-9393-1b9f12ee4d10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,070 [235] [DEBUG] [app] Ending request: urn:request:d4779f52-ab6f-459c-9393-1b9f12ee4d10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4779f52-ab6f-459c-9393-1b9f12ee4d10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:52,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:07:52,072 [235] [DEBUG] [app] Starting request: urn:request:1b276c2b-d6f0-4363-b281-51c9c6903001 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:07:52,072 [235] [DEBUG] [app] Ending request: urn:request:1b276c2b-d6f0-4363-b281-51c9c6903001 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1b276c2b-d6f0-4363-b281-51c9c6903001', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:07:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:07:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:07:52,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:07:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:07:52,087 [231] [DEBUG] [app] Ending request: urn:request:9c50c7a8-7773-4297-843e-f0924b258b69 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c50c7a8-7773-4297-843e-f0924b258b69', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:07:52,088 [236] [DEBUG] [app] Ending request: urn:request:f3765cfb-0beb-499e-93de-83c170c824f5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f3765cfb-0beb-499e-93de-83c170c824f5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:07:52,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:07:52,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) notificationworker stdout | 2023-01-17 05:07:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:07:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:01.636362+00:00 (in 5.989515 seconds) notificationworker stdout | 2023-01-17 05:07:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:05 UTC)" (scheduled at 2023-01-17 05:07:55.646354+00:00) notificationworker stdout | 2023-01-17 05:07:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:07:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 7, 55, 647234), True, datetime.datetime(2023, 1, 17, 5, 7, 55, 647234), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:07:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:07:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:07:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:05 UTC)" executed successfully builder stdout | 2023-01-17 05:07:56,745 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:07:56,763 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:07:56,763 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:07:56,774 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:07:57,638 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:07:57,659 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:07:58,533 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:07:59,004 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:07:59,062 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:07:59,438 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:07:59,582 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:08:00,716 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:08:00,720 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:08:00,722 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:08:00,724 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:08:00,726 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:08:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:05.646354+00:00 (in 4.009496 seconds) notificationworker stdout | 2023-01-17 05:08:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:01 UTC)" (scheduled at 2023-01-17 05:08:01.636362+00:00) notificationworker stdout | 2023-01-17 05:08:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:08:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:08:02,591 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:08:02,941 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:08:04,102 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:08:04,154 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:08:04,299 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:08:04,717 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:08:04,719 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:08:04,721 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:08:04,838 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} repositorygcworker stdout | 2023-01-17 05:08:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:08:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:19.346120+00:00 (in 14.408851 seconds) repositorygcworker stdout | 2023-01-17 05:08:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:04 UTC)" (scheduled at 2023-01-17 05:08:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:08:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:08:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 4, 937685), True, datetime.datetime(2023, 1, 17, 5, 8, 4, 937685), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:08:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:08:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:08:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:08:04,968 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:08:04,973 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:08:04,975 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:08:04,981 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:08:04,987 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:08:04,990 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:08:04,993 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:08:04,995 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:08:04,998 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:08:05,082 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:08:05,305 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:08:05,450 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:08:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:15.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 05:08:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:15 UTC)" (scheduled at 2023-01-17 05:08:05.646354+00:00) notificationworker stdout | 2023-01-17 05:08:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 5, 647238), True, datetime.datetime(2023, 1, 17, 5, 8, 5, 647238), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:08:05,814 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:08:06,358 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:08:06,465 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:08:06,629 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:08:06,776 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:06,788 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:06,789 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:06,799 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:07,060 [235] [DEBUG] [app] Starting request: urn:request:3744e34f-fdfa-442d-94fa-61730f376e76 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:07,060 [233] [DEBUG] [app] Starting request: urn:request:132cdfc9-a377-45cb-bec6-76d748ab57ed (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:07,064 [242] [DEBUG] [app] Starting request: urn:request:76f651bb-2d41-4207-a0ad-1d2a13ef5bbb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:07,065 [230] [DEBUG] [app] Starting request: urn:request:7cba900f-a589-40bd-a2e6-bb6f750c8d9c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:07,065 [242] [DEBUG] [app] Ending request: urn:request:76f651bb-2d41-4207-a0ad-1d2a13ef5bbb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76f651bb-2d41-4207-a0ad-1d2a13ef5bbb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:08:07,065 [230] [DEBUG] [app] Ending request: urn:request:7cba900f-a589-40bd-a2e6-bb6f750c8d9c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7cba900f-a589-40bd-a2e6-bb6f750c8d9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:08:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:08:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:07,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:07,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:07,069 [236] [DEBUG] [app] Starting request: urn:request:47ba1326-cf52-48ac-a3e8-e9584881d334 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:07,069 [235] [DEBUG] [app] Starting request: urn:request:28e87758-9e09-44e5-a414-8cf3c09c01da (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:07,070 [236] [DEBUG] [app] Ending request: urn:request:47ba1326-cf52-48ac-a3e8-e9584881d334 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47ba1326-cf52-48ac-a3e8-e9584881d334', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:07,070 [235] [DEBUG] [app] Ending request: urn:request:28e87758-9e09-44e5-a414-8cf3c09c01da (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28e87758-9e09-44e5-a414-8cf3c09c01da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:08:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:07,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:07,087 [233] [DEBUG] [app] Ending request: urn:request:132cdfc9-a377-45cb-bec6-76d748ab57ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:132cdfc9-a377-45cb-bec6-76d748ab57ed', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:07,087 [235] [DEBUG] [app] Ending request: urn:request:3744e34f-fdfa-442d-94fa-61730f376e76 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3744e34f-fdfa-442d-94fa-61730f376e76', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:07,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:08:07,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) buildlogsarchiver stdout | 2023-01-17 05:08:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:08:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:40.738568+00:00 (in 29.999526 seconds) buildlogsarchiver stdout | 2023-01-17 05:08:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:40 UTC)" (scheduled at 2023-01-17 05:08:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:08:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 8, 10, 739448), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:08:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:08:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:08:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:08:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:25.646354+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:08:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:25 UTC)" (scheduled at 2023-01-17 05:08:15.646354+00:00) notificationworker stdout | 2023-01-17 05:08:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 15, 647204), True, datetime.datetime(2023, 1, 17, 5, 8, 15, 647204), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:25 UTC)" executed successfully builder stdout | 2023-01-17 05:08:16,809 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:16,822 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:16,822 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:16,833 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:19.537472+00:00 (in 2.600041 seconds) securityworker stdout | 2023-01-17 05:08:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:46 UTC)" (scheduled at 2023-01-17 05:08:16.936988+00:00) securityworker stdout | 2023-01-17 05:08:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:08:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:08:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:04.936835+00:00 (in 45.590213 seconds) repositorygcworker stdout | 2023-01-17 05:08:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:19 UTC)" (scheduled at 2023-01-17 05:08:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:08:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:08:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:08:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:46.936988+00:00 (in 27.399043 seconds) securityworker stdout | 2023-01-17 05:08:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:49 UTC)" (scheduled at 2023-01-17 05:08:19.537472+00:00) securityworker stdout | 2023-01-17 05:08:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:08:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:08:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:08:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:08:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:08:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:08:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:08:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:08:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:51.348605+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 05:08:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:51 UTC)" (scheduled at 2023-01-17 05:08:21.348605+00:00) gcworker stdout | 2023-01-17 05:08:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722501349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:08:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:08:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:08:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:08:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:08:22,060 [236] [DEBUG] [app] Starting request: urn:request:3c966e3e-161a-4aa9-9709-7970711a8e8b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:22,060 [235] [DEBUG] [app] Starting request: urn:request:e8a3ef77-3756-4c6b-a010-ab98d8b0e9c9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:22,066 [243] [DEBUG] [app] Starting request: urn:request:0ccf1bb8-3443-48c6-9ded-dcfb99e623c7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:22,066 [230] [DEBUG] [app] Starting request: urn:request:63b27e76-385a-4366-a929-4265871143f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:22,066 [243] [DEBUG] [app] Ending request: urn:request:0ccf1bb8-3443-48c6-9ded-dcfb99e623c7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ccf1bb8-3443-48c6-9ded-dcfb99e623c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:08:22,067 [230] [DEBUG] [app] Ending request: urn:request:63b27e76-385a-4366-a929-4265871143f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:63b27e76-385a-4366-a929-4265871143f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:08:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:08:22,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:08:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:22,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:22,071 [231] [DEBUG] [app] Starting request: urn:request:61c7efdb-cc40-4f2d-8f58-953c08c17bf2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:22,072 [236] [DEBUG] [app] Starting request: urn:request:4088db1d-1a3f-44ad-8437-5f39a3c3ffc5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:22,072 [231] [DEBUG] [app] Ending request: urn:request:61c7efdb-cc40-4f2d-8f58-953c08c17bf2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61c7efdb-cc40-4f2d-8f58-953c08c17bf2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:22,072 [236] [DEBUG] [app] Ending request: urn:request:4088db1d-1a3f-44ad-8437-5f39a3c3ffc5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4088db1d-1a3f-44ad-8437-5f39a3c3ffc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:22,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:08:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:22,073 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:22,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:22,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:22,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:22,089 [235] [DEBUG] [app] Ending request: urn:request:e8a3ef77-3756-4c6b-a010-ab98d8b0e9c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e8a3ef77-3756-4c6b-a010-ab98d8b0e9c9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:22,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-web stdout | 2023-01-17 05:08:22,097 [236] [DEBUG] [app] Ending request: urn:request:3c966e3e-161a-4aa9-9709-7970711a8e8b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3c966e3e-161a-4aa9-9709-7970711a8e8b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:22,097 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:22,097 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.038 47 0.038) notificationworker stdout | 2023-01-17 05:08:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:35.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:08:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:35 UTC)" (scheduled at 2023-01-17 05:08:25.646354+00:00) notificationworker stdout | 2023-01-17 05:08:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 25, 647212), True, datetime.datetime(2023, 1, 17, 5, 8, 25, 647212), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:35 UTC)" executed successfully builder stdout | 2023-01-17 05:08:26,834 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:26,846 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:26,846 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:26,857 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:08:27,674 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:08:27,682 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:08:28,547 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:08:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:08:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:35.853396+00:00 (in 6.889264 seconds) exportactionlogsworker stdout | 2023-01-17 05:08:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:28 UTC)" (scheduled at 2023-01-17 05:08:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:08:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:08:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:28 UTC)" executed successfully builder stdout | 2023-01-17 05:08:29,022 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:08:29,083 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:08:29,462 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:08:29,603 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:08:30,723 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:08:30,726 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:08:30,729 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:08:30,731 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:08:30,733 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:08:32,599 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:08:32,976 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:08:34,134 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:08:34,174 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:08:34,334 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:08:34,724 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:08:34,727 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:08:34,729 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:08:34,873 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:08:34,977 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:08:34,980 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:08:34,983 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:08:34,988 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:08:34,993 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:08:34,996 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:08:34,999 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:08:35,002 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:08:35,004 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:08:35,097 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:08:35,340 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:08:35,460 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:08:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:36.635997+00:00 (in 0.989204 seconds) notificationworker stdout | 2023-01-17 05:08:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:45 UTC)" (scheduled at 2023-01-17 05:08:35.646354+00:00) notificationworker stdout | 2023-01-17 05:08:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 35, 647158), True, datetime.datetime(2023, 1, 17, 5, 8, 35, 647158), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:45 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:08:35,849 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:08:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:08:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:28.963662+00:00 (in 53.109783 seconds) exportactionlogsworker stdout | 2023-01-17 05:08:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:35 UTC)" (scheduled at 2023-01-17 05:08:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:08:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:08:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 35, 854302), True, datetime.datetime(2023, 1, 17, 5, 8, 35, 854302), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:08:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:08:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:08:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:35 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:08:36,394 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:08:36,500 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} notificationworker stdout | 2023-01-17 05:08:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:45.646354+00:00 (in 9.009936 seconds) notificationworker stdout | 2023-01-17 05:08:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:13:36 UTC)" (scheduled at 2023-01-17 05:08:36.635997+00:00) notificationworker stdout | 2023-01-17 05:08:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 8, 36, 636781), 'notification/%']) notificationworker stdout | 2023-01-17 05:08:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 8, 36, 636781), True, datetime.datetime(2023, 1, 17, 5, 8, 36, 636781), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:08:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 8, 36, 636781), True, datetime.datetime(2023, 1, 17, 5, 8, 36, 636781), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 8, 36, 636781), 'notification/%']) notificationworker stdout | 2023-01-17 05:08:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:13:36 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:08:36,660 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:08:36,867 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:36,879 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:36,879 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:36,890 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:37,059 [233] [DEBUG] [app] Starting request: urn:request:493c006b-c2c0-4403-bd4d-d2e513fb889a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:37,060 [235] [DEBUG] [app] Starting request: urn:request:e1bcb797-4a85-438b-bef7-9f5dd6e6a926 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:37,064 [230] [DEBUG] [app] Starting request: urn:request:519fbc8a-965a-410f-b2d9-fdf016dee43d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:37,065 [230] [DEBUG] [app] Ending request: urn:request:519fbc8a-965a-410f-b2d9-fdf016dee43d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:519fbc8a-965a-410f-b2d9-fdf016dee43d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:37,065 [242] [DEBUG] [app] Starting request: urn:request:25888f4b-0324-4513-96a1-9649fa62df84 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:08:37,065 [242] [DEBUG] [app] Ending request: urn:request:25888f4b-0324-4513-96a1-9649fa62df84 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:25888f4b-0324-4513-96a1-9649fa62df84', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:37,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:08:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:08:37,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:37,066 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:37,068 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:37,069 [231] [DEBUG] [app] Starting request: urn:request:27295e6f-e799-461d-b98f-0dc40de1b4a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:37,069 [231] [DEBUG] [app] Ending request: urn:request:27295e6f-e799-461d-b98f-0dc40de1b4a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27295e6f-e799-461d-b98f-0dc40de1b4a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:37,070 [236] [DEBUG] [app] Starting request: urn:request:053684ec-0c8f-4b73-ac9b-2ec7cf918e82 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:37,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:37,070 [236] [DEBUG] [app] Ending request: urn:request:053684ec-0c8f-4b73-ac9b-2ec7cf918e82 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:053684ec-0c8f-4b73-ac9b-2ec7cf918e82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:37,070 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:37,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:37,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:37,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:37,086 [235] [DEBUG] [app] Ending request: urn:request:e1bcb797-4a85-438b-bef7-9f5dd6e6a926 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1bcb797-4a85-438b-bef7-9f5dd6e6a926', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:37,086 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:08:37,089 [233] [DEBUG] [app] Ending request: urn:request:493c006b-c2c0-4403-bd4d-d2e513fb889a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:493c006b-c2c0-4403-bd4d-d2e513fb889a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:37,089 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 05:08:37,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:08:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:08:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:37.537407+00:00 (in 0.280332 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:08:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:37 UTC)" (scheduled at 2023-01-17 05:08:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:08:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:08:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 37, 257496), True, datetime.datetime(2023, 1, 17, 5, 8, 37, 257496), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:08:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:08:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:08:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:08:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:08:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:37.256652+00:00 (in 59.718873 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:08:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:37 UTC)" (scheduled at 2023-01-17 05:08:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:08:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:08:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:08:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:08:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:10.738568+00:00 (in 29.999517 seconds) buildlogsarchiver stdout | 2023-01-17 05:08:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:10 UTC)" (scheduled at 2023-01-17 05:08:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:08:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 8, 40, 739488), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:08:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:08:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:08:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:08:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:08:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:42.941536+00:00 (in 1.792791 seconds) namespacegcworker stdout | 2023-01-17 05:08:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:41 UTC)" (scheduled at 2023-01-17 05:08:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:08:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:08:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 41, 149197), True, datetime.datetime(2023, 1, 17, 5, 8, 41, 149197), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:08:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:08:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:08:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:08:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:08:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:41.148245+00:00 (in 58.206180 seconds) namespacegcworker stdout | 2023-01-17 05:08:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:42 UTC)" (scheduled at 2023-01-17 05:08:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:08:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:08:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:09:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:08:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:55.646354+00:00 (in 9.999552 seconds) notificationworker stdout | 2023-01-17 05:08:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:55 UTC)" (scheduled at 2023-01-17 05:08:45.646354+00:00) notificationworker stdout | 2023-01-17 05:08:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 45, 647179), True, datetime.datetime(2023, 1, 17, 5, 8, 45, 647179), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:08:55 UTC)" executed successfully builder stdout | 2023-01-17 05:08:46,891 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:46,904 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:46,904 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:46,915 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:08:49.537472+00:00 (in 2.600045 seconds) securityworker stdout | 2023-01-17 05:08:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:16 UTC)" (scheduled at 2023-01-17 05:08:46.936988+00:00) securityworker stdout | 2023-01-17 05:08:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:08:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:08:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:16.936988+00:00 (in 27.399080 seconds) securityworker stdout | 2023-01-17 05:08:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:19 UTC)" (scheduled at 2023-01-17 05:08:49.537472+00:00) securityworker stdout | 2023-01-17 05:08:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:08:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:08:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:08:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:08:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:08:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:08:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:08:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:08:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:08:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:08:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:08:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:08:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:08:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:08:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:08:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:21.348605+00:00 (in 29.999488 seconds) gcworker stdout | 2023-01-17 05:08:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:21 UTC)" (scheduled at 2023-01-17 05:08:51.348605+00:00) gcworker stdout | 2023-01-17 05:08:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722531349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:08:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:08:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:08:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:08:52,060 [236] [DEBUG] [app] Starting request: urn:request:f0c23c5d-dc30-4cd5-a914-363e3b88d4c2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:52,060 [235] [DEBUG] [app] Starting request: urn:request:309ec8b8-7982-4132-959d-ab82b17eff17 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:08:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:52,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:08:52,065 [241] [DEBUG] [app] Starting request: urn:request:324d27e1-29a9-40e4-aaa1-0a7f9771cccb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:52,065 [240] [DEBUG] [app] Starting request: urn:request:f548360a-8c2c-40b9-b750-01580bca4149 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:08:52,066 [241] [DEBUG] [app] Ending request: urn:request:324d27e1-29a9-40e4-aaa1-0a7f9771cccb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:324d27e1-29a9-40e4-aaa1-0a7f9771cccb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:52,066 [240] [DEBUG] [app] Ending request: urn:request:f548360a-8c2c-40b9-b750-01580bca4149 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f548360a-8c2c-40b9-b750-01580bca4149', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:08:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:08:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:08:52,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:08:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:08:52,071 [231] [DEBUG] [app] Starting request: urn:request:cca6a805-03dd-418f-a890-0eba5bd88737 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:52,071 [235] [DEBUG] [app] Starting request: urn:request:321c9a6c-37f1-44b2-a090-9780517e1a3e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:08:52,071 [231] [DEBUG] [app] Ending request: urn:request:cca6a805-03dd-418f-a890-0eba5bd88737 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cca6a805-03dd-418f-a890-0eba5bd88737', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:08:52,071 [235] [DEBUG] [app] Ending request: urn:request:321c9a6c-37f1-44b2-a090-9780517e1a3e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:321c9a6c-37f1-44b2-a090-9780517e1a3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:08:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:52,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:08:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:08:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:08:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:08:52,088 [235] [DEBUG] [app] Ending request: urn:request:309ec8b8-7982-4132-959d-ab82b17eff17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:309ec8b8-7982-4132-959d-ab82b17eff17', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:52,088 [236] [DEBUG] [app] Ending request: urn:request:f0c23c5d-dc30-4cd5-a914-363e3b88d4c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0c23c5d-dc30-4cd5-a914-363e3b88d4c2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:08:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:08:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:08:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:08:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:08:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:01.636362+00:00 (in 5.989562 seconds) notificationworker stdout | 2023-01-17 05:08:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:05 UTC)" (scheduled at 2023-01-17 05:08:55.646354+00:00) notificationworker stdout | 2023-01-17 05:08:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:08:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 8, 55, 647173), True, datetime.datetime(2023, 1, 17, 5, 8, 55, 647173), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:08:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:08:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:08:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:05 UTC)" executed successfully builder stdout | 2023-01-17 05:08:56,915 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:08:56,927 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:08:56,927 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:08:56,938 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:08:57,697 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:08:57,710 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:08:58,583 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:08:59,054 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:08:59,118 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:08:59,498 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:08:59,639 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:09:00,731 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:09:00,734 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:09:00,737 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:09:00,739 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:09:00,742 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:09:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:05.646354+00:00 (in 4.009516 seconds) notificationworker stdout | 2023-01-17 05:09:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:01 UTC)" (scheduled at 2023-01-17 05:09:01.636362+00:00) notificationworker stdout | 2023-01-17 05:09:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:09:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:09:02,632 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:09:02,996 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:09:04,161 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:09:04,209 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:09:04,368 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:09:04,733 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:09:04,736 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:09:04,738 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:09:04,908 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} repositorygcworker stdout | 2023-01-17 05:09:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:09:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:19.346120+00:00 (in 14.408848 seconds) repositorygcworker stdout | 2023-01-17 05:09:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:04 UTC)" (scheduled at 2023-01-17 05:09:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:09:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:09:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 4, 937729), True, datetime.datetime(2023, 1, 17, 5, 9, 4, 937729), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:09:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:09:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:09:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:09:04,985 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:09:04,987 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:09:04,990 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:09:04,994 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:09:05,000 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:09:05,003 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:09:05,006 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:09:05,008 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:09:05,013 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:09:05,132 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:09:05,376 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:09:05,488 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:09:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:15.646354+00:00 (in 9.999525 seconds) notificationworker stdout | 2023-01-17 05:09:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:15 UTC)" (scheduled at 2023-01-17 05:09:05.646354+00:00) notificationworker stdout | 2023-01-17 05:09:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 5, 647264), True, datetime.datetime(2023, 1, 17, 5, 9, 5, 647264), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:09:05,856 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:09:06,401 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:09:06,534 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:09:06,688 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:09:06,948 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:06,961 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:06,961 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:06,972 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:07,059 [233] [DEBUG] [app] Starting request: urn:request:0e382dad-13b6-4784-b9ac-a7f9997f31a0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:09:07,060 [231] [DEBUG] [app] Starting request: urn:request:6e8bad61-60ce-427c-a741-f70c14a82273 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:09:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:07,065 [230] [DEBUG] [app] Starting request: urn:request:ff2a01cf-78c0-4487-b196-e8a2de5ca395 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,065 [241] [DEBUG] [app] Starting request: urn:request:165e8979-b50f-4552-809a-55ed39b12dc3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,065 [230] [DEBUG] [app] Ending request: urn:request:ff2a01cf-78c0-4487-b196-e8a2de5ca395 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff2a01cf-78c0-4487-b196-e8a2de5ca395', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,065 [241] [DEBUG] [app] Ending request: urn:request:165e8979-b50f-4552-809a-55ed39b12dc3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:165e8979-b50f-4552-809a-55ed39b12dc3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:09:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:09:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:07,070 [236] [DEBUG] [app] Starting request: urn:request:0a789c18-a30a-4637-9b2f-5a6ea83c7004 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:07,071 [236] [DEBUG] [app] Ending request: urn:request:0a789c18-a30a-4637-9b2f-5a6ea83c7004 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a789c18-a30a-4637-9b2f-5a6ea83c7004', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:07,070 [231] [DEBUG] [app] Starting request: urn:request:6d1d349a-66c0-4be1-8a69-2accd28726a2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:07,071 [231] [DEBUG] [app] Ending request: urn:request:6d1d349a-66c0-4be1-8a69-2accd28726a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d1d349a-66c0-4be1-8a69-2accd28726a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:09:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:07,088 [231] [DEBUG] [app] Ending request: urn:request:6e8bad61-60ce-427c-a741-f70c14a82273 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e8bad61-60ce-427c-a741-f70c14a82273', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:07,088 [233] [DEBUG] [app] Ending request: urn:request:0e382dad-13b6-4784-b9ac-a7f9997f31a0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e382dad-13b6-4784-b9ac-a7f9997f31a0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:09:07,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:09:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" buildlogsarchiver stdout | 2023-01-17 05:09:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:09:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:40.738568+00:00 (in 29.999524 seconds) buildlogsarchiver stdout | 2023-01-17 05:09:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:40 UTC)" (scheduled at 2023-01-17 05:09:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:09:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 9, 10, 739392), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:09:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:09:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:09:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:09:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:25.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 05:09:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:25 UTC)" (scheduled at 2023-01-17 05:09:15.646354+00:00) notificationworker stdout | 2023-01-17 05:09:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 15, 647260), True, datetime.datetime(2023, 1, 17, 5, 9, 15, 647260), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:09:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:19.537472+00:00 (in 2.600050 seconds) securityworker stdout | 2023-01-17 05:09:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:46 UTC)" (scheduled at 2023-01-17 05:09:16.936988+00:00) securityworker stdout | 2023-01-17 05:09:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:46 UTC)" executed successfully builder stdout | 2023-01-17 05:09:16,982 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:16,993 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:16,993 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:17,004 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:09:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:09:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:04.936835+00:00 (in 45.590194 seconds) repositorygcworker stdout | 2023-01-17 05:09:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:19 UTC)" (scheduled at 2023-01-17 05:09:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:09:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:09:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:09:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:46.936988+00:00 (in 27.399038 seconds) securityworker stdout | 2023-01-17 05:09:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:49 UTC)" (scheduled at 2023-01-17 05:09:19.537472+00:00) securityworker stdout | 2023-01-17 05:09:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:09:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:09:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:09:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:09:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:09:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:09:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:09:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:09:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:51.348605+00:00 (in 29.999538 seconds) gcworker stdout | 2023-01-17 05:09:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:51 UTC)" (scheduled at 2023-01-17 05:09:21.348605+00:00) gcworker stdout | 2023-01-17 05:09:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722561349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:09:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:09:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:09:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:09:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:09:22,060 [231] [DEBUG] [app] Starting request: urn:request:9dea2058-bf92-4c5e-b1f3-b64829064f1a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:09:22,060 [235] [DEBUG] [app] Starting request: urn:request:b3c0fb98-6d89-4956-9697-aacba80f24ff (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:09:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:22,065 [241] [DEBUG] [app] Starting request: urn:request:0f8001f0-0764-482e-84d8-e238afb186dc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:22,065 [241] [DEBUG] [app] Ending request: urn:request:0f8001f0-0764-482e-84d8-e238afb186dc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f8001f0-0764-482e-84d8-e238afb186dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:22,065 [243] [DEBUG] [app] Starting request: urn:request:52a3cb0b-33e7-4648-b8a5-aa17dac5b379 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:09:22,066 [243] [DEBUG] [app] Ending request: urn:request:52a3cb0b-33e7-4648-b8a5-aa17dac5b379 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:52a3cb0b-33e7-4648-b8a5-aa17dac5b379', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:09:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:09:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:22,070 [233] [DEBUG] [app] Starting request: urn:request:b1047a2b-31d0-4aa2-b606-58795bfde032 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:22,071 [231] [DEBUG] [app] Starting request: urn:request:92d7ed21-f92e-4304-8c3b-4e1edef25867 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:22,071 [233] [DEBUG] [app] Ending request: urn:request:b1047a2b-31d0-4aa2-b606-58795bfde032 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b1047a2b-31d0-4aa2-b606-58795bfde032', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:22,071 [231] [DEBUG] [app] Ending request: urn:request:92d7ed21-f92e-4304-8c3b-4e1edef25867 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92d7ed21-f92e-4304-8c3b-4e1edef25867', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:22,088 [231] [DEBUG] [app] Ending request: urn:request:9dea2058-bf92-4c5e-b1f3-b64829064f1a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9dea2058-bf92-4c5e-b1f3-b64829064f1a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:22,088 [235] [DEBUG] [app] Ending request: urn:request:b3c0fb98-6d89-4956-9697-aacba80f24ff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b3c0fb98-6d89-4956-9697-aacba80f24ff', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:09:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) notificationworker stdout | 2023-01-17 05:09:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:35.646354+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:09:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:35 UTC)" (scheduled at 2023-01-17 05:09:25.646354+00:00) notificationworker stdout | 2023-01-17 05:09:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 25, 647296), True, datetime.datetime(2023, 1, 17, 5, 9, 25, 647296), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:35 UTC)" executed successfully builder stdout | 2023-01-17 05:09:27,014 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:27,027 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:27,027 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:27,038 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:27,717 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:09:27,729 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:09:28,619 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:09:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:09:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:35.853396+00:00 (in 6.889261 seconds) exportactionlogsworker stdout | 2023-01-17 05:09:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:28 UTC)" (scheduled at 2023-01-17 05:09:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:09:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:09:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:28 UTC)" executed successfully builder stdout | 2023-01-17 05:09:29,090 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:09:29,131 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:09:29,521 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:09:29,646 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:09:30,738 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:09:30,741 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:09:30,744 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:09:30,746 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:09:30,748 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:09:32,649 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:09:33,011 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:09:34,192 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:09:34,218 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:09:34,386 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:09:34,741 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:09:34,743 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:09:34,745 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} chunkcleanupworker stdout | 2023-01-17 05:09:34,942 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:09:34,992 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:09:34,998 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:09:35,000 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:09:35,002 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:09:35,007 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:09:35,009 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:09:35,016 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:09:35,018 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:09:35,023 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:09:35,167 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:09:35,412 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:09:35,493 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:09:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:45.646354+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 05:09:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:45 UTC)" (scheduled at 2023-01-17 05:09:35.646354+00:00) notificationworker stdout | 2023-01-17 05:09:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 35, 647255), True, datetime.datetime(2023, 1, 17, 5, 9, 35, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:09:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:09:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:28.963662+00:00 (in 53.109762 seconds) exportactionlogsworker stdout | 2023-01-17 05:09:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:35 UTC)" (scheduled at 2023-01-17 05:09:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:09:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:09:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 35, 854359), True, datetime.datetime(2023, 1, 17, 5, 9, 35, 854359), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:09:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:09:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:09:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:09:35,884 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:09:36,437 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:09:36,566 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:09:36,723 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:09:37,048 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-web stdout | 2023-01-17 05:09:37,060 [231] [DEBUG] [app] Starting request: urn:request:a8bd5bc0-3d65-4c3e-968d-77afe5cf90d8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:09:37,060 [236] [DEBUG] [app] Starting request: urn:request:30ac4911-56dd-43a9-b763-5c637a46b44d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} builder stdout | 2023-01-17 05:09:37,061 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:37,061 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-web stdout | 2023-01-17 05:09:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:37,065 [230] [DEBUG] [app] Starting request: urn:request:5e916e6c-eb41-4203-bcc6-b91335a54b67 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:37,065 [242] [DEBUG] [app] Starting request: urn:request:e56b1608-0fe9-44c1-b223-5da0fedd5d12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:37,065 [230] [DEBUG] [app] Ending request: urn:request:5e916e6c-eb41-4203-bcc6-b91335a54b67 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5e916e6c-eb41-4203-bcc6-b91335a54b67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:09:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:09:37,066 [242] [DEBUG] [app] Ending request: urn:request:e56b1608-0fe9-44c1-b223-5da0fedd5d12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e56b1608-0fe9-44c1-b223-5da0fedd5d12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:09:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:09:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:37,070 [235] [DEBUG] [app] Starting request: urn:request:899b00ec-67e0-4ad7-b6b7-825f480a964b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:37,070 [231] [DEBUG] [app] Starting request: urn:request:e429e784-ef1e-4106-9b79-837aa1e4cbaa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:37,070 [235] [DEBUG] [app] Ending request: urn:request:899b00ec-67e0-4ad7-b6b7-825f480a964b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:899b00ec-67e0-4ad7-b6b7-825f480a964b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:37,070 [231] [DEBUG] [app] Ending request: urn:request:e429e784-ef1e-4106-9b79-837aa1e4cbaa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e429e784-ef1e-4106-9b79-837aa1e4cbaa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql builder stdout | 2023-01-17 05:09:37,072 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:37,086 [231] [DEBUG] [app] Ending request: urn:request:a8bd5bc0-3d65-4c3e-968d-77afe5cf90d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8bd5bc0-3d65-4c3e-968d-77afe5cf90d8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:37,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:09:37,088 [236] [DEBUG] [app] Ending request: urn:request:30ac4911-56dd-43a9-b763-5c637a46b44d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:30ac4911-56dd-43a9-b763-5c637a46b44d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:09:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:09:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:09:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:37.537407+00:00 (in 0.280351 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:09:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:37 UTC)" (scheduled at 2023-01-17 05:09:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:09:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:09:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 37, 257518), True, datetime.datetime(2023, 1, 17, 5, 9, 37, 257518), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:09:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:09:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:09:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:09:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:09:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:37.256652+00:00 (in 59.718818 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:09:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:37 UTC)" (scheduled at 2023-01-17 05:09:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:09:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:09:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:09:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:09:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:10.738568+00:00 (in 29.999546 seconds) buildlogsarchiver stdout | 2023-01-17 05:09:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:10 UTC)" (scheduled at 2023-01-17 05:09:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:09:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 9, 40, 739434), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:09:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:09:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:09:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:09:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:09:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:42.941536+00:00 (in 1.792748 seconds) namespacegcworker stdout | 2023-01-17 05:09:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:41 UTC)" (scheduled at 2023-01-17 05:09:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:09:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:09:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 41, 149198), True, datetime.datetime(2023, 1, 17, 5, 9, 41, 149198), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:09:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:09:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:09:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:09:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:09:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:41.148245+00:00 (in 58.206255 seconds) namespacegcworker stdout | 2023-01-17 05:09:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:42 UTC)" (scheduled at 2023-01-17 05:09:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:09:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:09:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:10:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:09:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:55.646354+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:09:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:55 UTC)" (scheduled at 2023-01-17 05:09:45.646354+00:00) notificationworker stdout | 2023-01-17 05:09:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 45, 647254), True, datetime.datetime(2023, 1, 17, 5, 9, 45, 647254), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:09:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:09:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:09:49.537472+00:00 (in 2.600017 seconds) securityworker stdout | 2023-01-17 05:09:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:16 UTC)" (scheduled at 2023-01-17 05:09:46.936988+00:00) securityworker stdout | 2023-01-17 05:09:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:16 UTC)" executed successfully builder stdout | 2023-01-17 05:09:47,079 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:47,091 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:47,091 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:47,102 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:09:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:16.936988+00:00 (in 27.399042 seconds) securityworker stdout | 2023-01-17 05:09:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:19 UTC)" (scheduled at 2023-01-17 05:09:49.537472+00:00) securityworker stdout | 2023-01-17 05:09:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:09:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:09:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:09:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:09:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:09:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:09:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:09:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:09:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:09:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:09:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:09:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:09:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:09:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:09:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:09:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:21.348605+00:00 (in 29.999552 seconds) gcworker stdout | 2023-01-17 05:09:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:21 UTC)" (scheduled at 2023-01-17 05:09:51.348605+00:00) gcworker stdout | 2023-01-17 05:09:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722591349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:09:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:09:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:09:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:09:52,060 [231] [DEBUG] [app] Starting request: urn:request:20a0747b-2d17-425b-b484-064752fc1fc9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:09:52,060 [235] [DEBUG] [app] Starting request: urn:request:3ae43006-d8ff-4fa7-955d-81f36d130f39 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:09:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:09:52,064 [230] [DEBUG] [app] Starting request: urn:request:764130ca-3305-4478-9815-58629619b91e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:52,065 [242] [DEBUG] [app] Starting request: urn:request:4b9cf091-602a-4803-89b6-1a4536dd01fd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:09:52,065 [230] [DEBUG] [app] Ending request: urn:request:764130ca-3305-4478-9815-58629619b91e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:764130ca-3305-4478-9815-58629619b91e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:09:52,065 [242] [DEBUG] [app] Ending request: urn:request:4b9cf091-602a-4803-89b6-1a4536dd01fd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4b9cf091-602a-4803-89b6-1a4536dd01fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:09:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:09:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:09:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:09:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:09:52,070 [236] [DEBUG] [app] Starting request: urn:request:a1e5bea0-1f43-4475-be2e-c3b492682ad7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:52,070 [233] [DEBUG] [app] Starting request: urn:request:575d2778-4378-4cbd-be8c-58017abc2243 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:09:52,071 [236] [DEBUG] [app] Ending request: urn:request:a1e5bea0-1f43-4475-be2e-c3b492682ad7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1e5bea0-1f43-4475-be2e-c3b492682ad7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:09:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:52,071 [233] [DEBUG] [app] Ending request: urn:request:575d2778-4378-4cbd-be8c-58017abc2243 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:575d2778-4378-4cbd-be8c-58017abc2243', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:09:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:09:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:09:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:09:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:09:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:09:52,087 [231] [DEBUG] [app] Ending request: urn:request:20a0747b-2d17-425b-b484-064752fc1fc9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20a0747b-2d17-425b-b484-064752fc1fc9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:52,087 [235] [DEBUG] [app] Ending request: urn:request:3ae43006-d8ff-4fa7-955d-81f36d130f39 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ae43006-d8ff-4fa7-955d-81f36d130f39', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:09:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:09:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:09:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:09:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:09:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:01.636362+00:00 (in 5.989517 seconds) notificationworker stdout | 2023-01-17 05:09:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:05 UTC)" (scheduled at 2023-01-17 05:09:55.646354+00:00) notificationworker stdout | 2023-01-17 05:09:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:09:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 9, 55, 647262), True, datetime.datetime(2023, 1, 17, 5, 9, 55, 647262), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:09:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:09:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:05 UTC)" executed successfully builder stdout | 2023-01-17 05:09:57,112 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:09:57,125 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:09:57,125 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:09:57,135 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:09:57,730 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:09:57,764 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:09:58,655 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:09:59,101 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:09:59,141 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:09:59,548 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:09:59,676 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:10:00,745 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:10:00,748 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:10:00,750 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:10:00,753 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:10:00,755 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:10:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:05.646354+00:00 (in 4.009500 seconds) notificationworker stdout | 2023-01-17 05:10:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:01 UTC)" (scheduled at 2023-01-17 05:10:01.636362+00:00) notificationworker stdout | 2023-01-17 05:10:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:10:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:10:02,685 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:10:03,037 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:10:04,209 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:10:04,245 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:10:04,421 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:10:04,748 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:10:04,751 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:10:04,754 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:10:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:10:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:19.346120+00:00 (in 14.408824 seconds) repositorygcworker stdout | 2023-01-17 05:10:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:04 UTC)" (scheduled at 2023-01-17 05:10:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:10:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:10:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 4, 937786), True, datetime.datetime(2023, 1, 17, 5, 10, 4, 937786), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:10:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:10:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:10:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:04 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:10:04,977 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:10:04,998 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:10:05,007 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:10:05,010 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:10:05,012 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:10:05,017 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:10:05,020 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:10:05,026 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:10:05,028 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:10:05,031 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:10:05,200 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:10:05,445 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:10:05,528 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:10:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:15.646354+00:00 (in 9.999508 seconds) notificationworker stdout | 2023-01-17 05:10:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:15 UTC)" (scheduled at 2023-01-17 05:10:05.646354+00:00) notificationworker stdout | 2023-01-17 05:10:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 5, 647205), True, datetime.datetime(2023, 1, 17, 5, 10, 5, 647205), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:10:05,920 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:10:06,443 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:10:06,601 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:10:06,758 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:10:07,060 [233] [DEBUG] [app] Starting request: urn:request:6bd8135f-f569-4e44-a9ca-ee27f80dd9f7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:07,060 [236] [DEBUG] [app] Starting request: urn:request:95318049-978d-4e5f-9270-dab0d61c3caa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:07,065 [230] [DEBUG] [app] Starting request: urn:request:ff0f9b8d-ccb1-45ed-b3a9-56c246a46798 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:07,065 [242] [DEBUG] [app] Starting request: urn:request:f9715b02-79b1-4229-9856-5afd85fe574e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:07,065 [230] [DEBUG] [app] Ending request: urn:request:ff0f9b8d-ccb1-45ed-b3a9-56c246a46798 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff0f9b8d-ccb1-45ed-b3a9-56c246a46798', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:07,066 [242] [DEBUG] [app] Ending request: urn:request:f9715b02-79b1-4229-9856-5afd85fe574e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9715b02-79b1-4229-9856-5afd85fe574e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:10:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:10:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:07,070 [231] [DEBUG] [app] Starting request: urn:request:1fbb3540-1ba6-46ac-a8a6-2acb199fddc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:07,071 [236] [DEBUG] [app] Starting request: urn:request:0344e3ad-d4da-4545-8bf5-c31ac9ec5089 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:07,071 [231] [DEBUG] [app] Ending request: urn:request:1fbb3540-1ba6-46ac-a8a6-2acb199fddc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1fbb3540-1ba6-46ac-a8a6-2acb199fddc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:07,071 [236] [DEBUG] [app] Ending request: urn:request:0344e3ad-d4da-4545-8bf5-c31ac9ec5089 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0344e3ad-d4da-4545-8bf5-c31ac9ec5089', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:07,088 [236] [DEBUG] [app] Ending request: urn:request:95318049-978d-4e5f-9270-dab0d61c3caa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:95318049-978d-4e5f-9270-dab0d61c3caa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:07,088 [233] [DEBUG] [app] Ending request: urn:request:6bd8135f-f569-4e44-a9ca-ee27f80dd9f7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6bd8135f-f569-4e44-a9ca-ee27f80dd9f7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:10:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:10:07,146 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:07,157 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:07,157 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:07,168 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:10:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:10:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:40.738568+00:00 (in 29.999557 seconds) buildlogsarchiver stdout | 2023-01-17 05:10:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:40 UTC)" (scheduled at 2023-01-17 05:10:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:10:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 10, 10, 739419), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:10:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:10:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:10:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:10:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:25.646354+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 05:10:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:25 UTC)" (scheduled at 2023-01-17 05:10:15.646354+00:00) notificationworker stdout | 2023-01-17 05:10:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 15, 647240), True, datetime.datetime(2023, 1, 17, 5, 10, 15, 647240), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:10:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:19.537472+00:00 (in 2.599995 seconds) securityworker stdout | 2023-01-17 05:10:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:46 UTC)" (scheduled at 2023-01-17 05:10:16.936988+00:00) securityworker stdout | 2023-01-17 05:10:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:46 UTC)" executed successfully builder stdout | 2023-01-17 05:10:17,178 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:17,191 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:17,191 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:17,202 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:10:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:10:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:04.936835+00:00 (in 45.590134 seconds) repositorygcworker stdout | 2023-01-17 05:10:19,347 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:19 UTC)" (scheduled at 2023-01-17 05:10:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:10:19,347 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:10:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:10:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:46.936988+00:00 (in 27.399083 seconds) securityworker stdout | 2023-01-17 05:10:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:49 UTC)" (scheduled at 2023-01-17 05:10:19.537472+00:00) securityworker stdout | 2023-01-17 05:10:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:10:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:10:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:10:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:10:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:10:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:10:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:10:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:10:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:51.348605+00:00 (in 29.999538 seconds) gcworker stdout | 2023-01-17 05:10:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:51 UTC)" (scheduled at 2023-01-17 05:10:21.348605+00:00) gcworker stdout | 2023-01-17 05:10:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722621349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:10:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:10:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:10:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:10:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:10:22,059 [231] [DEBUG] [app] Starting request: urn:request:78ab0f02-eb75-492d-88fe-1902d8c27d88 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:22,059 [235] [DEBUG] [app] Starting request: urn:request:1a6b7d53-ceb8-4bd0-9d0c-14b871c2d08d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:22,064 [230] [DEBUG] [app] Starting request: urn:request:9bc630f7-8a3a-4c0b-b1df-5263312a131e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:22,065 [243] [DEBUG] [app] Starting request: urn:request:7342714e-6ae2-4fb0-9258-ed79e0ce997a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:22,065 [230] [DEBUG] [app] Ending request: urn:request:9bc630f7-8a3a-4c0b-b1df-5263312a131e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9bc630f7-8a3a-4c0b-b1df-5263312a131e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:10:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:10:22,065 [243] [DEBUG] [app] Ending request: urn:request:7342714e-6ae2-4fb0-9258-ed79e0ce997a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7342714e-6ae2-4fb0-9258-ed79e0ce997a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:10:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:10:22,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:22,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:22,069 [236] [DEBUG] [app] Starting request: urn:request:32eed797-ae10-4529-8fac-f64fcccbdfa3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:22,070 [236] [DEBUG] [app] Ending request: urn:request:32eed797-ae10-4529-8fac-f64fcccbdfa3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32eed797-ae10-4529-8fac-f64fcccbdfa3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:22,070 [231] [DEBUG] [app] Starting request: urn:request:9d159b0e-66b6-4a0c-a2b6-2a4d9c7c8987 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:22,070 [231] [DEBUG] [app] Ending request: urn:request:9d159b0e-66b6-4a0c-a2b6-2a4d9c7c8987 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9d159b0e-66b6-4a0c-a2b6-2a4d9c7c8987', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:22,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:22,087 [231] [DEBUG] [app] Ending request: urn:request:78ab0f02-eb75-492d-88fe-1902d8c27d88 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78ab0f02-eb75-492d-88fe-1902d8c27d88', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:22,087 [235] [DEBUG] [app] Ending request: urn:request:1a6b7d53-ceb8-4bd0-9d0c-14b871c2d08d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1a6b7d53-ceb8-4bd0-9d0c-14b871c2d08d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:22,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:10:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:10:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:35.646354+00:00 (in 9.999503 seconds) notificationworker stdout | 2023-01-17 05:10:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:35 UTC)" (scheduled at 2023-01-17 05:10:25.646354+00:00) notificationworker stdout | 2023-01-17 05:10:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 25, 647256), True, datetime.datetime(2023, 1, 17, 5, 10, 25, 647256), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:35 UTC)" executed successfully builder stdout | 2023-01-17 05:10:27,212 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:27,224 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:27,225 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:27,235 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:27,743 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:10:27,799 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:10:28,691 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:10:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:10:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:35.853396+00:00 (in 6.889275 seconds) exportactionlogsworker stdout | 2023-01-17 05:10:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:28 UTC)" (scheduled at 2023-01-17 05:10:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:10:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:10:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:28 UTC)" executed successfully builder stdout | 2023-01-17 05:10:29,136 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:10:29,175 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:10:29,585 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:10:29,711 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:10:30,753 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:10:30,756 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:10:30,759 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:10:30,761 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:10:30,764 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:10:32,720 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:10:33,049 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:10:34,244 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:10:34,252 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:10:34,439 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:10:34,756 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:10:34,758 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:10:34,760 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:10:35,004 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} chunkcleanupworker stdout | 2023-01-17 05:10:35,010 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:10:35,015 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:10:35,018 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:10:35,021 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:10:35,027 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:10:35,029 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:10:35,035 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:10:35,037 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:10:35,040 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} manifestbackfillworker stdout | 2023-01-17 05:10:35,206 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:10:35,453 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:10:35,546 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:10:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:45.646354+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:10:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:45 UTC)" (scheduled at 2023-01-17 05:10:35.646354+00:00) notificationworker stdout | 2023-01-17 05:10:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 35, 647204), True, datetime.datetime(2023, 1, 17, 5, 10, 35, 647204), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:10:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:10:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:28.963662+00:00 (in 53.109800 seconds) exportactionlogsworker stdout | 2023-01-17 05:10:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:35 UTC)" (scheduled at 2023-01-17 05:10:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:10:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:10:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 35, 854222), True, datetime.datetime(2023, 1, 17, 5, 10, 35, 854222), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:10:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:10:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:10:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:10:35,952 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:10:36,479 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:10:36,636 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:10:36,794 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:10:37,060 [235] [DEBUG] [app] Starting request: urn:request:fb39a0d0-5827-4c17-aaef-693e93d98054 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:37,060 [231] [DEBUG] [app] Starting request: urn:request:bd7626a3-8f51-4ea0-b535-5c1ab28df128 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:37,065 [241] [DEBUG] [app] Starting request: urn:request:afa40ba1-678d-4b67-b6b7-257b4fb8316b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:37,066 [243] [DEBUG] [app] Starting request: urn:request:cda24719-6ced-4b42-b66f-645589ea4257 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:37,066 [241] [DEBUG] [app] Ending request: urn:request:afa40ba1-678d-4b67-b6b7-257b4fb8316b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:afa40ba1-678d-4b67-b6b7-257b4fb8316b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:10:37,066 [243] [DEBUG] [app] Ending request: urn:request:cda24719-6ced-4b42-b66f-645589ea4257 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cda24719-6ced-4b42-b66f-645589ea4257', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:10:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:10:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:10:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:37,071 [236] [DEBUG] [app] Starting request: urn:request:abbe30b8-bd30-4a8f-b654-e0523045379b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:37,071 [235] [DEBUG] [app] Starting request: urn:request:ff211749-04c8-431a-9c7b-35583589d547 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:37,071 [236] [DEBUG] [app] Ending request: urn:request:abbe30b8-bd30-4a8f-b654-e0523045379b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:abbe30b8-bd30-4a8f-b654-e0523045379b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:37,071 [235] [DEBUG] [app] Ending request: urn:request:ff211749-04c8-431a-9c7b-35583589d547 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff211749-04c8-431a-9c7b-35583589d547', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:37,088 [235] [DEBUG] [app] Ending request: urn:request:fb39a0d0-5827-4c17-aaef-693e93d98054 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb39a0d0-5827-4c17-aaef-693e93d98054', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:10:37,088 [231] [DEBUG] [app] Ending request: urn:request:bd7626a3-8f51-4ea0-b535-5c1ab28df128 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd7626a3-8f51-4ea0-b535-5c1ab28df128', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:10:37,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:10:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:10:37,245 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue securityscanningnotificationworker stdout | 2023-01-17 05:10:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:10:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:37.537407+00:00 (in 0.280308 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:10:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:37 UTC)" (scheduled at 2023-01-17 05:10:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:10:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. builder stdout | 2023-01-17 05:10:37,257 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:37,258 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics securityscanningnotificationworker stdout | 2023-01-17 05:10:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 37, 257578), True, datetime.datetime(2023, 1, 17, 5, 10, 37, 257578), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:10:37,268 [81] [DEBUG] [workers.queueworker] No more work. builder stdout | 2023-01-17 05:10:37,268 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:10:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:10:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:10:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:10:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:23.537048+00:00 (in 45.999091 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:10:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:37 UTC)" (scheduled at 2023-01-17 05:10:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:10:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:10:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:10:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:10:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:10.738568+00:00 (in 29.999530 seconds) buildlogsarchiver stdout | 2023-01-17 05:10:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:10 UTC)" (scheduled at 2023-01-17 05:10:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:10:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 10, 40, 739410), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:10:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:10:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:10:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:10:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:10:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:42.941536+00:00 (in 1.792827 seconds) namespacegcworker stdout | 2023-01-17 05:10:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:41 UTC)" (scheduled at 2023-01-17 05:10:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:10:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:10:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 41, 149118), True, datetime.datetime(2023, 1, 17, 5, 10, 41, 149118), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:10:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:10:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:10:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:10:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:10:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:41.148245+00:00 (in 58.206220 seconds) namespacegcworker stdout | 2023-01-17 05:10:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:42 UTC)" (scheduled at 2023-01-17 05:10:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:10:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:10:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:11:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:10:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:55.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:10:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:55 UTC)" (scheduled at 2023-01-17 05:10:45.646354+00:00) notificationworker stdout | 2023-01-17 05:10:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 45, 647204), True, datetime.datetime(2023, 1, 17, 5, 10, 45, 647204), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:10:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:10:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:10:49.537472+00:00 (in 2.600014 seconds) securityworker stdout | 2023-01-17 05:10:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:16 UTC)" (scheduled at 2023-01-17 05:10:46.936988+00:00) securityworker stdout | 2023-01-17 05:10:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:16 UTC)" executed successfully builder stdout | 2023-01-17 05:10:47,278 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:47,293 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:47,293 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:47,304 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:10:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:16.936988+00:00 (in 27.399000 seconds) securityworker stdout | 2023-01-17 05:10:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:19 UTC)" (scheduled at 2023-01-17 05:10:49.537472+00:00) securityworker stdout | 2023-01-17 05:10:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:10:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:10:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:10:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:10:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:10:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:10:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:10:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:10:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:10:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:10:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:10:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:10:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:10:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:10:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:10:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:21.348605+00:00 (in 29.999557 seconds) gcworker stdout | 2023-01-17 05:10:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:21 UTC)" (scheduled at 2023-01-17 05:10:51.348605+00:00) gcworker stdout | 2023-01-17 05:10:51,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:10:51,360 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722651359, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:10:51,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:10:51,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:10:51,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:10:52,060 [231] [DEBUG] [app] Starting request: urn:request:51bf8316-cc57-4587-a441-95a535084cd5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:52,060 [235] [DEBUG] [app] Starting request: urn:request:39f67311-89e9-44e0-8e48-90eea68a126a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:10:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:10:52,065 [230] [DEBUG] [app] Starting request: urn:request:a841e027-c54c-470c-a4bc-36a7f01ee9e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,065 [243] [DEBUG] [app] Starting request: urn:request:3b81d6ae-b8db-4507-9907-678e38afa30b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,066 [230] [DEBUG] [app] Ending request: urn:request:a841e027-c54c-470c-a4bc-36a7f01ee9e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a841e027-c54c-470c-a4bc-36a7f01ee9e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,066 [243] [DEBUG] [app] Ending request: urn:request:3b81d6ae-b8db-4507-9907-678e38afa30b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b81d6ae-b8db-4507-9907-678e38afa30b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:10:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:10:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:10:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:10:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:10:52,071 [236] [DEBUG] [app] Starting request: urn:request:88c1066e-c74a-43b5-840a-80f3b9421f74 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:52,072 [233] [DEBUG] [app] Starting request: urn:request:df3fd4f9-f983-4edf-b47b-b85547207802 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:10:52,072 [236] [DEBUG] [app] Ending request: urn:request:88c1066e-c74a-43b5-840a-80f3b9421f74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:88c1066e-c74a-43b5-840a-80f3b9421f74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:10:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:52,072 [233] [DEBUG] [app] Ending request: urn:request:df3fd4f9-f983-4edf-b47b-b85547207802 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df3fd4f9-f983-4edf-b47b-b85547207802', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:10:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:10:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:10:52,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:10:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:52,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:10:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:10:52,089 [235] [DEBUG] [app] Ending request: urn:request:39f67311-89e9-44e0-8e48-90eea68a126a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39f67311-89e9-44e0-8e48-90eea68a126a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:52,089 [231] [DEBUG] [app] Ending request: urn:request:51bf8316-cc57-4587-a441-95a535084cd5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:51bf8316-cc57-4587-a441-95a535084cd5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:10:52,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:10:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) gunicorn-web stdout | 2023-01-17 05:10:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:10:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:10:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:01.636362+00:00 (in 5.989547 seconds) notificationworker stdout | 2023-01-17 05:10:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:05 UTC)" (scheduled at 2023-01-17 05:10:55.646354+00:00) notificationworker stdout | 2023-01-17 05:10:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:10:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 10, 55, 647240), True, datetime.datetime(2023, 1, 17, 5, 10, 55, 647240), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:10:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:10:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:05 UTC)" executed successfully builder stdout | 2023-01-17 05:10:57,314 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:10:57,326 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:10:57,326 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:10:57,337 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:10:57,750 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:10:57,808 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:10:58,727 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:10:59,172 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:10:59,210 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:10:59,620 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:10:59,723 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:11:00,761 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:11:00,763 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:11:00,766 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:11:00,769 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:11:00,771 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:11:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:05.646354+00:00 (in 4.009539 seconds) notificationworker stdout | 2023-01-17 05:11:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:01 UTC)" (scheduled at 2023-01-17 05:11:01.636362+00:00) notificationworker stdout | 2023-01-17 05:11:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:11:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:11:02,756 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:11:03,082 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:11:04,257 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:11:04,280 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:11:04,474 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:11:04,763 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:11:04,765 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:11:04,768 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:11:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:11:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:17.345601+00:00 (in 12.408244 seconds) repositorygcworker stdout | 2023-01-17 05:11:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:04 UTC)" (scheduled at 2023-01-17 05:11:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:11:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:11:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 4, 937845), True, datetime.datetime(2023, 1, 17, 5, 11, 4, 937845), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:11:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:11:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:11:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:11:05,016 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} chunkcleanupworker stdout | 2023-01-17 05:11:05,019 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:11:05,021 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:11:05,026 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:11:05,030 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:11:05,032 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:11:05,038 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:11:05,046 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:11:05,048 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:11:05,050 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} manifestbackfillworker stdout | 2023-01-17 05:11:05,241 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:11:05,486 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:11:05,561 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:11:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:15.646354+00:00 (in 9.999452 seconds) notificationworker stdout | 2023-01-17 05:11:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:15 UTC)" (scheduled at 2023-01-17 05:11:05.646354+00:00) notificationworker stdout | 2023-01-17 05:11:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 5, 647301), True, datetime.datetime(2023, 1, 17, 5, 11, 5, 647301), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:11:05,988 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:11:06,485 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:11:06,646 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:11:06,829 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:11:07,060 [233] [DEBUG] [app] Starting request: urn:request:6e6bcd6a-1996-40a3-9ff3-4a317f6b4730 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:07,060 [231] [DEBUG] [app] Starting request: urn:request:6f65e4b9-e0b3-45df-aeda-9972a1499b99 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:07,065 [243] [DEBUG] [app] Starting request: urn:request:e92375d5-faee-4595-b16d-13f3c1f587a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:07,066 [242] [DEBUG] [app] Starting request: urn:request:5d15fac2-5657-4d31-adca-81945b88898a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:07,066 [243] [DEBUG] [app] Ending request: urn:request:e92375d5-faee-4595-b16d-13f3c1f587a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e92375d5-faee-4595-b16d-13f3c1f587a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:11:07,066 [242] [DEBUG] [app] Ending request: urn:request:5d15fac2-5657-4d31-adca-81945b88898a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d15fac2-5657-4d31-adca-81945b88898a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:11:07,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:11:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:07,070 [236] [DEBUG] [app] Starting request: urn:request:4ad5dd2a-049c-424f-8799-ffa95064024b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:07,071 [236] [DEBUG] [app] Ending request: urn:request:4ad5dd2a-049c-424f-8799-ffa95064024b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4ad5dd2a-049c-424f-8799-ffa95064024b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:07,071 [235] [DEBUG] [app] Starting request: urn:request:7a255c17-eaa5-448c-8bfd-399d9e057eb8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:07,071 [235] [DEBUG] [app] Ending request: urn:request:7a255c17-eaa5-448c-8bfd-399d9e057eb8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a255c17-eaa5-448c-8bfd-399d9e057eb8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:11:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:07,082 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:07,082 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:07,091 [233] [DEBUG] [app] Ending request: urn:request:6e6bcd6a-1996-40a3-9ff3-4a317f6b4730 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e6bcd6a-1996-40a3-9ff3-4a317f6b4730', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:07,091 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:07,091 [231] [DEBUG] [app] Ending request: urn:request:6f65e4b9-e0b3-45df-aeda-9972a1499b99 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f65e4b9-e0b3-45df-aeda-9972a1499b99', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:07,091 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) gunicorn-web stdout | 2023-01-17 05:11:07,091 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:11:07,092 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:11:07,347 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:07,359 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:07,359 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:07,370 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:11:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:11:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:40.738568+00:00 (in 29.999530 seconds) buildlogsarchiver stdout | 2023-01-17 05:11:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:40 UTC)" (scheduled at 2023-01-17 05:11:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:11:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 11, 10, 739433), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:11:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:11:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:11:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:11:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:25.646354+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 05:11:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:25 UTC)" (scheduled at 2023-01-17 05:11:15.646354+00:00) notificationworker stdout | 2023-01-17 05:11:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 15, 647254), True, datetime.datetime(2023, 1, 17, 5, 11, 15, 647254), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:11:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:19.537472+00:00 (in 2.600002 seconds) securityworker stdout | 2023-01-17 05:11:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:46 UTC)" (scheduled at 2023-01-17 05:11:16.936988+00:00) securityworker stdout | 2023-01-17 05:11:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:11:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:11:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:19.346120+00:00 (in 2.000058 seconds) repositorygcworker stdout | 2023-01-17 05:11:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:17 UTC)" (scheduled at 2023-01-17 05:11:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:11:17,346 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 11, 17, 346463), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:11:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 17, 346463), True, datetime.datetime(2023, 1, 17, 5, 11, 17, 346463), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:11:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 17, 346463), True, datetime.datetime(2023, 1, 17, 5, 11, 17, 346463), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 11, 17, 346463), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:11:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:11:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:17 UTC)" executed successfully builder stdout | 2023-01-17 05:11:17,375 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:17,387 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:17,387 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:17,397 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:11:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:11:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:04.936835+00:00 (in 45.590213 seconds) repositorygcworker stdout | 2023-01-17 05:11:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:19 UTC)" (scheduled at 2023-01-17 05:11:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:11:19,347 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:11:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:11:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:46.936988+00:00 (in 27.399043 seconds) securityworker stdout | 2023-01-17 05:11:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:49 UTC)" (scheduled at 2023-01-17 05:11:19.537472+00:00) securityworker stdout | 2023-01-17 05:11:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:11:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:11:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:11:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:11:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:11:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:11:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:11:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:11:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:51.348605+00:00 (in 29.999431 seconds) gcworker stdout | 2023-01-17 05:11:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:51 UTC)" (scheduled at 2023-01-17 05:11:21.348605+00:00) gcworker stdout | 2023-01-17 05:11:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722681349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:11:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:11:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:11:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:11:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:11:22,060 [235] [DEBUG] [app] Starting request: urn:request:25e23604-58a4-49d6-8cc7-bf414c1e636e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:22,060 [231] [DEBUG] [app] Starting request: urn:request:374c9ba1-6061-4289-a98a-73452128049c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:22,065 [230] [DEBUG] [app] Starting request: urn:request:0d507ef2-1bef-4567-811e-bd5243a7ca21 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:22,066 [243] [DEBUG] [app] Starting request: urn:request:455a7877-a30e-44f6-ba60-1775fc802dd3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:22,066 [230] [DEBUG] [app] Ending request: urn:request:0d507ef2-1bef-4567-811e-bd5243a7ca21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d507ef2-1bef-4567-811e-bd5243a7ca21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:22,066 [243] [DEBUG] [app] Ending request: urn:request:455a7877-a30e-44f6-ba60-1775fc802dd3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:455a7877-a30e-44f6-ba60-1775fc802dd3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:11:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:22,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:22,071 [236] [DEBUG] [app] Starting request: urn:request:b020ecde-6ccb-4c9c-b102-e3f41fcc9592 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:22,071 [233] [DEBUG] [app] Starting request: urn:request:0177fe38-1c71-4c47-91d9-1b9f8d3f115c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:22,071 [236] [DEBUG] [app] Ending request: urn:request:b020ecde-6ccb-4c9c-b102-e3f41fcc9592 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b020ecde-6ccb-4c9c-b102-e3f41fcc9592', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:22,072 [233] [DEBUG] [app] Ending request: urn:request:0177fe38-1c71-4c47-91d9-1b9f8d3f115c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0177fe38-1c71-4c47-91d9-1b9f8d3f115c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:11:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:22,089 [235] [DEBUG] [app] Ending request: urn:request:25e23604-58a4-49d6-8cc7-bf414c1e636e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:25e23604-58a4-49d6-8cc7-bf414c1e636e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:22,089 [231] [DEBUG] [app] Ending request: urn:request:374c9ba1-6061-4289-a98a-73452128049c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:374c9ba1-6061-4289-a98a-73452128049c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:11:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:11:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:11:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:37.256652+00:00 (in 13.719110 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:11:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:23 UTC)" (scheduled at 2023-01-17 05:11:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:11:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 11, 23, 537908), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:11:23,549 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 23, 537908), True, datetime.datetime(2023, 1, 17, 5, 11, 23, 537908), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:11:23,553 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 23, 537908), True, datetime.datetime(2023, 1, 17, 5, 11, 23, 537908), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 11, 23, 537908), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:11:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:11:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:11:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:35.646354+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 05:11:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:35 UTC)" (scheduled at 2023-01-17 05:11:25.646354+00:00) notificationworker stdout | 2023-01-17 05:11:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 25, 647280), True, datetime.datetime(2023, 1, 17, 5, 11, 25, 647280), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:35 UTC)" executed successfully builder stdout | 2023-01-17 05:11:27,408 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:27,420 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:27,420 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:27,431 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:27,786 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:11:27,840 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:11:28,740 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:11:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:11:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:35.853396+00:00 (in 6.889268 seconds) exportactionlogsworker stdout | 2023-01-17 05:11:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:28 UTC)" (scheduled at 2023-01-17 05:11:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:11:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:11:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:28 UTC)" executed successfully builder stdout | 2023-01-17 05:11:29,204 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:11:29,237 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:11:29,640 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:11:29,729 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:11:30,768 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:11:30,770 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:11:30,773 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:11:30,775 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:11:30,778 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:11:32,792 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:11:33,114 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:11:34,274 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:11:34,299 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:11:34,489 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:11:34,770 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:11:34,773 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:11:34,775 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:11:35,026 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:11:35,032 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:11:35,034 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:11:35,038 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:11:35,040 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:11:35,047 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} chunkcleanupworker stdout | 2023-01-17 05:11:35,049 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:11:35,054 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:11:35,056 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:11:35,059 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} manifestbackfillworker stdout | 2023-01-17 05:11:35,276 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:11:35,495 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:11:35,582 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:11:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:45.646354+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 05:11:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:45 UTC)" (scheduled at 2023-01-17 05:11:35.646354+00:00) notificationworker stdout | 2023-01-17 05:11:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 35, 647257), True, datetime.datetime(2023, 1, 17, 5, 11, 35, 647257), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:11:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:11:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:54.963413+00:00 (in 19.109531 seconds) exportactionlogsworker stdout | 2023-01-17 05:11:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:35 UTC)" (scheduled at 2023-01-17 05:11:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:11:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:11:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 35, 854247), True, datetime.datetime(2023, 1, 17, 5, 11, 35, 854247), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:11:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:11:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:11:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:11:36,023 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:11:36,521 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:11:36,672 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:11:36,864 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:11:37,060 [236] [DEBUG] [app] Starting request: urn:request:d72c15b5-1e0d-4025-a8c5-0d3c5b5b2616 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:37,060 [235] [DEBUG] [app] Starting request: urn:request:c4514c91-71d6-40bd-b729-6d1302feadd8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:37,065 [230] [DEBUG] [app] Starting request: urn:request:ca5dcc1e-ad74-4006-bae8-e1a8a966af1c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:37,065 [243] [DEBUG] [app] Starting request: urn:request:d8e4db41-1b44-45dc-85db-393165727644 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:37,065 [230] [DEBUG] [app] Ending request: urn:request:ca5dcc1e-ad74-4006-bae8-e1a8a966af1c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ca5dcc1e-ad74-4006-bae8-e1a8a966af1c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:37,065 [243] [DEBUG] [app] Ending request: urn:request:d8e4db41-1b44-45dc-85db-393165727644 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8e4db41-1b44-45dc-85db-393165727644', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:11:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:11:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:37,070 [233] [DEBUG] [app] Starting request: urn:request:c3f04f8a-72d6-4d9d-8d02-82cb3fff6b13 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:37,070 [236] [DEBUG] [app] Starting request: urn:request:7475643b-ffa3-4b51-8837-f26eaab74edc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:37,070 [233] [DEBUG] [app] Ending request: urn:request:c3f04f8a-72d6-4d9d-8d02-82cb3fff6b13 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c3f04f8a-72d6-4d9d-8d02-82cb3fff6b13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:37,071 [236] [DEBUG] [app] Ending request: urn:request:7475643b-ffa3-4b51-8837-f26eaab74edc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7475643b-ffa3-4b51-8837-f26eaab74edc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:37,087 [236] [DEBUG] [app] Ending request: urn:request:d72c15b5-1e0d-4025-a8c5-0d3c5b5b2616 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d72c15b5-1e0d-4025-a8c5-0d3c5b5b2616', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:37,087 [235] [DEBUG] [app] Ending request: urn:request:c4514c91-71d6-40bd-b729-6d1302feadd8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4514c91-71d6-40bd-b729-6d1302feadd8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:37,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:11:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) securityscanningnotificationworker stdout | 2023-01-17 05:11:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:11:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:37.537407+00:00 (in 0.280268 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:11:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:37 UTC)" (scheduled at 2023-01-17 05:11:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:11:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:11:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 37, 257524), True, datetime.datetime(2023, 1, 17, 5, 11, 37, 257524), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:11:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:11:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:11:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:37 UTC)" executed successfully builder stdout | 2023-01-17 05:11:37,441 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:37,453 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:37,453 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:37,464 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:11:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:11:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:37.256652+00:00 (in 59.718838 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:11:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:37 UTC)" (scheduled at 2023-01-17 05:11:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:11:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:11:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:37 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:11:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:11:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:10.738568+00:00 (in 29.999528 seconds) buildlogsarchiver stdout | 2023-01-17 05:11:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:10 UTC)" (scheduled at 2023-01-17 05:11:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:11:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 11, 40, 739431), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:11:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:11:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:11:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:11:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:11:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:42.941536+00:00 (in 1.792787 seconds) namespacegcworker stdout | 2023-01-17 05:11:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:41 UTC)" (scheduled at 2023-01-17 05:11:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:11:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:11:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 41, 149148), True, datetime.datetime(2023, 1, 17, 5, 11, 41, 149148), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:11:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:11:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:11:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:11:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:11:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:06.941185+00:00 (in 23.999162 seconds) namespacegcworker stdout | 2023-01-17 05:11:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:42 UTC)" (scheduled at 2023-01-17 05:11:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:11:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:11:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:12:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:11:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:55.646354+00:00 (in 9.999495 seconds) notificationworker stdout | 2023-01-17 05:11:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:55 UTC)" (scheduled at 2023-01-17 05:11:45.646354+00:00) notificationworker stdout | 2023-01-17 05:11:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 45, 647258), True, datetime.datetime(2023, 1, 17, 5, 11, 45, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:11:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:11:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:11:49.537472+00:00 (in 2.600031 seconds) securityworker stdout | 2023-01-17 05:11:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:16 UTC)" (scheduled at 2023-01-17 05:11:46.936988+00:00) securityworker stdout | 2023-01-17 05:11:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:16 UTC)" executed successfully builder stdout | 2023-01-17 05:11:47,474 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:47,486 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:47,486 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:47,497 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:11:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:16.936988+00:00 (in 27.399041 seconds) securityworker stdout | 2023-01-17 05:11:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:19 UTC)" (scheduled at 2023-01-17 05:11:49.537472+00:00) securityworker stdout | 2023-01-17 05:11:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:11:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:11:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:11:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:11:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:11:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:11:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:11:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:11:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:11:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:11:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:11:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:11:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:11:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:11:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:11:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:21.348605+00:00 (in 29.999494 seconds) gcworker stdout | 2023-01-17 05:11:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:21 UTC)" (scheduled at 2023-01-17 05:11:51.348605+00:00) gcworker stdout | 2023-01-17 05:11:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722711349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:11:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:11:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:11:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:11:52,060 [236] [DEBUG] [app] Starting request: urn:request:21856560-db1e-47bd-852d-9a1ce2149b1e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:52,060 [231] [DEBUG] [app] Starting request: urn:request:b8a95c0b-3e20-48da-baf3-047ae7ac8df0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:11:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:11:52,065 [241] [DEBUG] [app] Starting request: urn:request:54b8b081-6a4e-471b-ab20-325b170228ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:52,065 [242] [DEBUG] [app] Starting request: urn:request:0a8c40be-c66c-4fe8-a81a-9c86517e43a6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:11:52,066 [241] [DEBUG] [app] Ending request: urn:request:54b8b081-6a4e-471b-ab20-325b170228ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54b8b081-6a4e-471b-ab20-325b170228ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:52,066 [242] [DEBUG] [app] Ending request: urn:request:0a8c40be-c66c-4fe8-a81a-9c86517e43a6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a8c40be-c66c-4fe8-a81a-9c86517e43a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:11:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:11:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:11:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:11:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:11:52,071 [233] [DEBUG] [app] Starting request: urn:request:6b8955cc-c9aa-4fa6-9d58-11f4da928b18 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:52,071 [235] [DEBUG] [app] Starting request: urn:request:679b37ec-94bb-48c4-b7a7-0facc41a72cb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:11:52,071 [233] [DEBUG] [app] Ending request: urn:request:6b8955cc-c9aa-4fa6-9d58-11f4da928b18 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b8955cc-c9aa-4fa6-9d58-11f4da928b18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:11:52,071 [235] [DEBUG] [app] Ending request: urn:request:679b37ec-94bb-48c4-b7a7-0facc41a72cb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:679b37ec-94bb-48c4-b7a7-0facc41a72cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:11:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:11:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:11:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:11:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:11:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:11:52,088 [231] [DEBUG] [app] Ending request: urn:request:b8a95c0b-3e20-48da-baf3-047ae7ac8df0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b8a95c0b-3e20-48da-baf3-047ae7ac8df0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:52,088 [236] [DEBUG] [app] Ending request: urn:request:21856560-db1e-47bd-852d-9a1ce2149b1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21856560-db1e-47bd-852d-9a1ce2149b1e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:11:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:11:52,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:11:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:11:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 05:11:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:11:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:28.963662+00:00 (in 33.999809 seconds) exportactionlogsworker stdout | 2023-01-17 05:11:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:54 UTC)" (scheduled at 2023-01-17 05:11:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:11:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 11, 54, 964222), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:11:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 54, 964222), True, datetime.datetime(2023, 1, 17, 5, 11, 54, 964222), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:11:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 11, 54, 964222), True, datetime.datetime(2023, 1, 17, 5, 11, 54, 964222), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 11, 54, 964222), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:11:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:11:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:16:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:11:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:11:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:01.636362+00:00 (in 5.989569 seconds) notificationworker stdout | 2023-01-17 05:11:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:05 UTC)" (scheduled at 2023-01-17 05:11:55.646354+00:00) notificationworker stdout | 2023-01-17 05:11:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:11:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 11, 55, 647212), True, datetime.datetime(2023, 1, 17, 5, 11, 55, 647212), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:11:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:11:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:05 UTC)" executed successfully builder stdout | 2023-01-17 05:11:57,507 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:11:57,519 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:11:57,519 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:11:57,529 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:11:57,793 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:11:57,862 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:11:58,776 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:11:59,238 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:11:59,272 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:11:59,676 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:11:59,748 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:12:00,776 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:12:00,778 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:12:00,782 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:12:00,784 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:12:00,786 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:12:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:05.646354+00:00 (in 4.009525 seconds) notificationworker stdout | 2023-01-17 05:12:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:01 UTC)" (scheduled at 2023-01-17 05:12:01.636362+00:00) notificationworker stdout | 2023-01-17 05:12:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:12:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:12:02,828 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:12:03,149 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:12:04,310 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:12:04,318 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:12:04,525 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:12:04,776 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:12:04,780 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:12:04,782 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:12:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:12:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:19.346120+00:00 (in 14.408794 seconds) repositorygcworker stdout | 2023-01-17 05:12:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:04 UTC)" (scheduled at 2023-01-17 05:12:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:12:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:12:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 4, 937726), True, datetime.datetime(2023, 1, 17, 5, 12, 4, 937726), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:12:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:12:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:12:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:12:05,036 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:12:05,039 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:12:05,043 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:12:05,046 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:12:05,049 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:12:05,054 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:12:05,060 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} chunkcleanupworker stdout | 2023-01-17 05:12:05,062 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-registry stdout | 2023-01-17 05:12:05,064 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:12:05,066 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} manifestbackfillworker stdout | 2023-01-17 05:12:05,312 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:12:05,531 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:12:05,601 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:12:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:15.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:12:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:15 UTC)" (scheduled at 2023-01-17 05:12:05.646354+00:00) notificationworker stdout | 2023-01-17 05:12:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 5, 647191), True, datetime.datetime(2023, 1, 17, 5, 12, 5, 647191), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:12:06,038 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:12:06,557 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:12:06,685 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:12:06,870 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} namespacegcworker stdout | 2023-01-17 05:12:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:12:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:41.148245+00:00 (in 34.206612 seconds) namespacegcworker stdout | 2023-01-17 05:12:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:17:06 UTC)" (scheduled at 2023-01-17 05:12:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:12:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 12, 6, 941999), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:12:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 12, 6, 941999), True, datetime.datetime(2023, 1, 17, 5, 12, 6, 941999), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:12:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 12, 6, 941999), True, datetime.datetime(2023, 1, 17, 5, 12, 6, 941999), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 12, 6, 941999), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:12:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:12:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:17:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:12:07,059 [233] [DEBUG] [app] Starting request: urn:request:4cb4980f-5412-4339-bc8a-dcfc30473363 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:07,059 [231] [DEBUG] [app] Starting request: urn:request:10b3a76f-6221-4cee-8557-2b7a79c2cd6e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:07,065 [243] [DEBUG] [app] Starting request: urn:request:f0ec2401-9a94-4ecd-8db2-2a49c61d697d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:07,065 [241] [DEBUG] [app] Starting request: urn:request:ef4a7fac-7c57-4b16-add3-72512ec7a2d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:07,065 [243] [DEBUG] [app] Ending request: urn:request:f0ec2401-9a94-4ecd-8db2-2a49c61d697d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0ec2401-9a94-4ecd-8db2-2a49c61d697d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:12:07,065 [241] [DEBUG] [app] Ending request: urn:request:ef4a7fac-7c57-4b16-add3-72512ec7a2d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef4a7fac-7c57-4b16-add3-72512ec7a2d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:12:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:12:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:12:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:07,070 [236] [DEBUG] [app] Starting request: urn:request:1e2409a7-8de5-4ee6-8543-e7496eb00fe8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:07,070 [235] [DEBUG] [app] Starting request: urn:request:a2e7c8fb-bdc2-43f0-8299-aad7ddf4674e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:07,070 [236] [DEBUG] [app] Ending request: urn:request:1e2409a7-8de5-4ee6-8543-e7496eb00fe8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e2409a7-8de5-4ee6-8543-e7496eb00fe8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:07,071 [235] [DEBUG] [app] Ending request: urn:request:a2e7c8fb-bdc2-43f0-8299-aad7ddf4674e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a2e7c8fb-bdc2-43f0-8299-aad7ddf4674e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:07,087 [231] [DEBUG] [app] Ending request: urn:request:10b3a76f-6221-4cee-8557-2b7a79c2cd6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10b3a76f-6221-4cee-8557-2b7a79c2cd6e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:07,087 [233] [DEBUG] [app] Ending request: urn:request:4cb4980f-5412-4339-bc8a-dcfc30473363 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4cb4980f-5412-4339-bc8a-dcfc30473363', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:07,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:12:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:12:07,540 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:07,552 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:07,552 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:07,563 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:12:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:12:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:40.738568+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 05:12:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:40 UTC)" (scheduled at 2023-01-17 05:12:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:12:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 12, 10, 739454), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:12:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:12:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:12:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:12:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:25.646354+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 05:12:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:25 UTC)" (scheduled at 2023-01-17 05:12:15.646354+00:00) notificationworker stdout | 2023-01-17 05:12:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 15, 647231), True, datetime.datetime(2023, 1, 17, 5, 12, 15, 647231), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:12:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:19.537472+00:00 (in 2.599978 seconds) securityworker stdout | 2023-01-17 05:12:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:46 UTC)" (scheduled at 2023-01-17 05:12:16.936988+00:00) securityworker stdout | 2023-01-17 05:12:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:16,953 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:16,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:46 UTC)" executed successfully builder stdout | 2023-01-17 05:12:17,573 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:17,586 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:17,586 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:17,596 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:12:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:12:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:04.936835+00:00 (in 45.590338 seconds) repositorygcworker stdout | 2023-01-17 05:12:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:19 UTC)" (scheduled at 2023-01-17 05:12:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:12:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:12:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:12:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:46.936988+00:00 (in 27.399061 seconds) securityworker stdout | 2023-01-17 05:12:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:49 UTC)" (scheduled at 2023-01-17 05:12:19.537472+00:00) securityworker stdout | 2023-01-17 05:12:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:12:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:12:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:12:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:12:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:12:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:12:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:12:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:12:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:51.348605+00:00 (in 29.999493 seconds) gcworker stdout | 2023-01-17 05:12:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:51 UTC)" (scheduled at 2023-01-17 05:12:21.348605+00:00) gcworker stdout | 2023-01-17 05:12:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722741349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:12:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:12:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:12:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:12:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:12:22,060 [231] [DEBUG] [app] Starting request: urn:request:ba4ed10b-f12c-4576-8542-284b3c451e50 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:22,060 [233] [DEBUG] [app] Starting request: urn:request:73b2850d-7b48-4eea-8552-01eb57977cc6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:22,065 [230] [DEBUG] [app] Starting request: urn:request:16c1579b-1cbd-4244-8ff3-ca883ec02ed6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,065 [242] [DEBUG] [app] Starting request: urn:request:7d86bf74-b78a-4fb0-bb79-14128f9d6eea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,065 [230] [DEBUG] [app] Ending request: urn:request:16c1579b-1cbd-4244-8ff3-ca883ec02ed6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:16c1579b-1cbd-4244-8ff3-ca883ec02ed6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,066 [242] [DEBUG] [app] Ending request: urn:request:7d86bf74-b78a-4fb0-bb79-14128f9d6eea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7d86bf74-b78a-4fb0-bb79-14128f9d6eea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:12:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:12:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:22,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:22,070 [236] [DEBUG] [app] Starting request: urn:request:e525bb32-8285-41f4-bebb-8b3d117732df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:22,070 [235] [DEBUG] [app] Starting request: urn:request:56d31bf4-53fb-41a0-beb9-04d18c28a369 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:22,071 [236] [DEBUG] [app] Ending request: urn:request:e525bb32-8285-41f4-bebb-8b3d117732df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e525bb32-8285-41f4-bebb-8b3d117732df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:22,071 [235] [DEBUG] [app] Ending request: urn:request:56d31bf4-53fb-41a0-beb9-04d18c28a369 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56d31bf4-53fb-41a0-beb9-04d18c28a369', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:12:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:12:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:22,088 [231] [DEBUG] [app] Ending request: urn:request:ba4ed10b-f12c-4576-8542-284b3c451e50 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba4ed10b-f12c-4576-8542-284b3c451e50', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:22,088 [233] [DEBUG] [app] Ending request: urn:request:73b2850d-7b48-4eea-8552-01eb57977cc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:73b2850d-7b48-4eea-8552-01eb57977cc6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:12:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 05:12:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:35.646354+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 05:12:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:35 UTC)" (scheduled at 2023-01-17 05:12:25.646354+00:00) notificationworker stdout | 2023-01-17 05:12:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 25, 647154), True, datetime.datetime(2023, 1, 17, 5, 12, 25, 647154), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:35 UTC)" executed successfully builder stdout | 2023-01-17 05:12:27,606 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:27,619 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:27,619 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:27,629 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:27,825 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:12:27,892 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:12:28,812 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:12:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:12:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:35.853396+00:00 (in 6.889247 seconds) exportactionlogsworker stdout | 2023-01-17 05:12:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:28 UTC)" (scheduled at 2023-01-17 05:12:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:12:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:12:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:28 UTC)" executed successfully builder stdout | 2023-01-17 05:12:29,274 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:12:29,283 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:12:29,696 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:12:29,781 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:12:30,782 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:12:30,785 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:12:30,787 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:12:30,791 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:12:30,793 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:12:32,864 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:12:33,165 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:12:34,334 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:12:34,337 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:12:34,549 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:12:34,783 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:12:34,786 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:12:34,788 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:12:35,045 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:12:35,048 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:12:35,050 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:12:35,053 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:12:35,056 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:12:35,061 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:12:35,066 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:12:35,070 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:12:35,073 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} chunkcleanupworker stdout | 2023-01-17 05:12:35,096 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:12:35,318 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:12:35,548 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:12:35,621 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:12:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:45.646354+00:00 (in 9.999589 seconds) notificationworker stdout | 2023-01-17 05:12:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:45 UTC)" (scheduled at 2023-01-17 05:12:35.646354+00:00) notificationworker stdout | 2023-01-17 05:12:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 35, 647176), True, datetime.datetime(2023, 1, 17, 5, 12, 35, 647176), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:12:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:12:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:28.963662+00:00 (in 53.109759 seconds) exportactionlogsworker stdout | 2023-01-17 05:12:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:35 UTC)" (scheduled at 2023-01-17 05:12:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:12:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:12:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 35, 854261), True, datetime.datetime(2023, 1, 17, 5, 12, 35, 854261), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:12:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:12:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:12:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:12:36,067 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:12:36,566 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:12:36,701 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:12:36,905 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:12:37,060 [231] [DEBUG] [app] Starting request: urn:request:ab336b91-5fcc-4961-b457-6e397c9e5e49 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:37,060 [235] [DEBUG] [app] Starting request: urn:request:c8e4a816-b034-4979-96f3-282113b1ed8f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:37,065 [241] [DEBUG] [app] Starting request: urn:request:efc7879a-89cc-4e0b-87fa-a7e15cf8b339 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:37,065 [242] [DEBUG] [app] Starting request: urn:request:12a5716b-55e0-4f86-a0d2-aa2f1eeae0fc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:37,066 [241] [DEBUG] [app] Ending request: urn:request:efc7879a-89cc-4e0b-87fa-a7e15cf8b339 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:efc7879a-89cc-4e0b-87fa-a7e15cf8b339', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:12:37,066 [242] [DEBUG] [app] Ending request: urn:request:12a5716b-55e0-4f86-a0d2-aa2f1eeae0fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:12a5716b-55e0-4f86-a0d2-aa2f1eeae0fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:12:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:12:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:12:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:37,071 [233] [DEBUG] [app] Starting request: urn:request:b306355b-c8c7-41fa-aea0-7c76b00dd010 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:37,071 [235] [DEBUG] [app] Starting request: urn:request:fe5126ac-5445-423c-be72-1f0c76b3a1e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:37,072 [233] [DEBUG] [app] Ending request: urn:request:b306355b-c8c7-41fa-aea0-7c76b00dd010 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b306355b-c8c7-41fa-aea0-7c76b00dd010', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:37,072 [235] [DEBUG] [app] Ending request: urn:request:fe5126ac-5445-423c-be72-1f0c76b3a1e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe5126ac-5445-423c-be72-1f0c76b3a1e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:12:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:37,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:37,089 [235] [DEBUG] [app] Ending request: urn:request:c8e4a816-b034-4979-96f3-282113b1ed8f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c8e4a816-b034-4979-96f3-282113b1ed8f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:37,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:37,089 [231] [DEBUG] [app] Ending request: urn:request:ab336b91-5fcc-4961-b457-6e397c9e5e49 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab336b91-5fcc-4961-b457-6e397c9e5e49', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:12:37,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) gunicorn-web stdout | 2023-01-17 05:12:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:12:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:12:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:37.537407+00:00 (in 0.280342 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:12:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:37 UTC)" (scheduled at 2023-01-17 05:12:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:12:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:12:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 37, 257509), True, datetime.datetime(2023, 1, 17, 5, 12, 37, 257509), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:12:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:12:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:12:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:12:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:12:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:37.256652+00:00 (in 59.718849 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:12:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:37 UTC)" (scheduled at 2023-01-17 05:12:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:12:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:12:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:37 UTC)" executed successfully builder stdout | 2023-01-17 05:12:37,639 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:37,651 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:37,651 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:37,662 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:12:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:12:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:10.738568+00:00 (in 29.999566 seconds) buildlogsarchiver stdout | 2023-01-17 05:12:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:10 UTC)" (scheduled at 2023-01-17 05:12:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:12:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 12, 40, 739425), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:12:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:12:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:12:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:12:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:12:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:42.941536+00:00 (in 1.792805 seconds) namespacegcworker stdout | 2023-01-17 05:12:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:41 UTC)" (scheduled at 2023-01-17 05:12:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:12:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:12:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 41, 149132), True, datetime.datetime(2023, 1, 17, 5, 12, 41, 149132), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:12:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:12:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:12:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:12:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:12:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:41.148245+00:00 (in 58.206213 seconds) namespacegcworker stdout | 2023-01-17 05:12:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:42 UTC)" (scheduled at 2023-01-17 05:12:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:12:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:12:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:13:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:12:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:55.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:12:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:55 UTC)" (scheduled at 2023-01-17 05:12:45.646354+00:00) notificationworker stdout | 2023-01-17 05:12:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 45, 647198), True, datetime.datetime(2023, 1, 17, 5, 12, 45, 647198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:12:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:12:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:12:49.537472+00:00 (in 2.600050 seconds) securityworker stdout | 2023-01-17 05:12:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:16 UTC)" (scheduled at 2023-01-17 05:12:46.936988+00:00) securityworker stdout | 2023-01-17 05:12:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:46,962 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:46,965 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:46,965 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:16 UTC)" executed successfully builder stdout | 2023-01-17 05:12:47,672 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:47,685 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:47,685 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:47,695 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:12:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:16.936988+00:00 (in 27.399080 seconds) securityworker stdout | 2023-01-17 05:12:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:19 UTC)" (scheduled at 2023-01-17 05:12:49.537472+00:00) securityworker stdout | 2023-01-17 05:12:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:12:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:12:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:12:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:12:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:12:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:12:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:12:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:12:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:12:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:12:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:12:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:12:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:12:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:12:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:12:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:21.348605+00:00 (in 29.999520 seconds) gcworker stdout | 2023-01-17 05:12:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:21 UTC)" (scheduled at 2023-01-17 05:12:51.348605+00:00) gcworker stdout | 2023-01-17 05:12:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722771349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:12:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:12:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:12:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:12:52,060 [231] [DEBUG] [app] Starting request: urn:request:c263e121-f39e-40cc-a636-e948e7f5c6f2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:52,060 [236] [DEBUG] [app] Starting request: urn:request:6bf4cbd8-449f-46db-b012-bb7f89ea9fe4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:12:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:12:52,064 [242] [DEBUG] [app] Starting request: urn:request:0248640e-da0f-4d8e-9b3c-0d757cb85846 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:52,064 [243] [DEBUG] [app] Starting request: urn:request:153eef04-d588-4b4f-8b15-04a91f0e03bc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:12:52,065 [242] [DEBUG] [app] Ending request: urn:request:0248640e-da0f-4d8e-9b3c-0d757cb85846 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0248640e-da0f-4d8e-9b3c-0d757cb85846', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:12:52,065 [243] [DEBUG] [app] Ending request: urn:request:153eef04-d588-4b4f-8b15-04a91f0e03bc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:153eef04-d588-4b4f-8b15-04a91f0e03bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:12:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:12:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:52,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:12:52,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:12:52,070 [235] [DEBUG] [app] Starting request: urn:request:2323b3a6-a1e1-4fbb-9129-e50b7c47bffc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:52,070 [236] [DEBUG] [app] Starting request: urn:request:d37f0dec-1560-416a-b505-42e67e34bc3f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:12:52,070 [235] [DEBUG] [app] Ending request: urn:request:2323b3a6-a1e1-4fbb-9129-e50b7c47bffc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2323b3a6-a1e1-4fbb-9129-e50b7c47bffc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:12:52,070 [236] [DEBUG] [app] Ending request: urn:request:d37f0dec-1560-416a-b505-42e67e34bc3f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d37f0dec-1560-416a-b505-42e67e34bc3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:12:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:12:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:12:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:12:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:12:52,088 [236] [DEBUG] [app] Ending request: urn:request:6bf4cbd8-449f-46db-b012-bb7f89ea9fe4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6bf4cbd8-449f-46db-b012-bb7f89ea9fe4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:52,088 [231] [DEBUG] [app] Ending request: urn:request:c263e121-f39e-40cc-a636-e948e7f5c6f2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c263e121-f39e-40cc-a636-e948e7f5c6f2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:12:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:12:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:12:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 05:12:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:12:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:01.636362+00:00 (in 5.989537 seconds) notificationworker stdout | 2023-01-17 05:12:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:05 UTC)" (scheduled at 2023-01-17 05:12:55.646354+00:00) notificationworker stdout | 2023-01-17 05:12:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:12:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 12, 55, 647253), True, datetime.datetime(2023, 1, 17, 5, 12, 55, 647253), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:12:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:12:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:05 UTC)" executed successfully builder stdout | 2023-01-17 05:12:57,705 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:12:57,718 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:12:57,719 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:12:57,729 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:12:57,861 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:12:57,900 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:12:58,839 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:12:59,302 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:12:59,305 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:12:59,730 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:12:59,811 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:13:00,790 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:13:00,793 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:13:00,795 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:13:00,798 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:13:00,801 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:13:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:05.646354+00:00 (in 4.009521 seconds) notificationworker stdout | 2023-01-17 05:13:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:01 UTC)" (scheduled at 2023-01-17 05:13:01.636362+00:00) notificationworker stdout | 2023-01-17 05:13:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:13:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:13:02,899 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:13:03,190 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:13:04,340 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:13:04,348 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:13:04,585 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:13:04,789 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:13:04,792 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:13:04,795 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:13:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:13:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:19.346120+00:00 (in 14.408837 seconds) repositorygcworker stdout | 2023-01-17 05:13:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:04 UTC)" (scheduled at 2023-01-17 05:13:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:13:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:13:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 4, 937759), True, datetime.datetime(2023, 1, 17, 5, 13, 4, 937759), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:13:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:13:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:13:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:13:05,053 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:13:05,056 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:13:05,059 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:13:05,061 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:13:05,064 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:13:05,067 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:13:05,072 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:13:05,079 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:13:05,082 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:13:05,131 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:13:05,353 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:13:05,572 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:13:05,638 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:13:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:15.646354+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 05:13:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:15 UTC)" (scheduled at 2023-01-17 05:13:05.646354+00:00) notificationworker stdout | 2023-01-17 05:13:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 5, 647250), True, datetime.datetime(2023, 1, 17, 5, 13, 5, 647250), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:15 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:13:06,102 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:13:06,602 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:13:06,736 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:13:06,940 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:13:07,059 [236] [DEBUG] [app] Starting request: urn:request:460bf458-4364-4044-8b8a-69a83f3a77bd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:07,060 [233] [DEBUG] [app] Starting request: urn:request:5df765ca-cd2f-4e7d-905d-9a709bed8b66 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:07,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:07,064 [243] [DEBUG] [app] Starting request: urn:request:45f25684-e7d8-472f-a467-24523545840e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:07,064 [241] [DEBUG] [app] Starting request: urn:request:95befafe-ac92-4508-9e9b-58b8f7a4060d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:07,065 [243] [DEBUG] [app] Ending request: urn:request:45f25684-e7d8-472f-a467-24523545840e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:45f25684-e7d8-472f-a467-24523545840e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:07,065 [241] [DEBUG] [app] Ending request: urn:request:95befafe-ac92-4508-9e9b-58b8f7a4060d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:95befafe-ac92-4508-9e9b-58b8f7a4060d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:13:07,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:07,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:07,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:13:07,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,066 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:07,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:07,070 [231] [DEBUG] [app] Starting request: urn:request:ee26da4e-0a39-4938-ae0b-bcc1552d7398 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:07,070 [236] [DEBUG] [app] Starting request: urn:request:641ffc78-b694-4338-a539-7f12da9a0186 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:07,070 [231] [DEBUG] [app] Ending request: urn:request:ee26da4e-0a39-4938-ae0b-bcc1552d7398 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee26da4e-0a39-4938-ae0b-bcc1552d7398', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:07,070 [236] [DEBUG] [app] Ending request: urn:request:641ffc78-b694-4338-a539-7f12da9a0186 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:641ffc78-b694-4338-a539-7f12da9a0186', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:13:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:07,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:07,092 [236] [DEBUG] [app] Ending request: urn:request:460bf458-4364-4044-8b8a-69a83f3a77bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:460bf458-4364-4044-8b8a-69a83f3a77bd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:07,092 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:07,093 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.034 47 0.034) gunicorn-web stdout | 2023-01-17 05:13:07,093 [233] [DEBUG] [app] Ending request: urn:request:5df765ca-cd2f-4e7d-905d-9a709bed8b66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5df765ca-cd2f-4e7d-905d-9a709bed8b66', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:07,093 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:07,094 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) builder stdout | 2023-01-17 05:13:07,739 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:07,752 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:07,752 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:07,762 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:13:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:13:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:40.738568+00:00 (in 29.999563 seconds) buildlogsarchiver stdout | 2023-01-17 05:13:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:40 UTC)" (scheduled at 2023-01-17 05:13:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:13:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 13, 10, 739442), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:13:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:13:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:13:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:13:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:25.646354+00:00 (in 9.999501 seconds) notificationworker stdout | 2023-01-17 05:13:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:25 UTC)" (scheduled at 2023-01-17 05:13:15.646354+00:00) notificationworker stdout | 2023-01-17 05:13:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 15, 647298), True, datetime.datetime(2023, 1, 17, 5, 13, 15, 647298), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:13:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:19.537472+00:00 (in 2.600019 seconds) securityworker stdout | 2023-01-17 05:13:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:46 UTC)" (scheduled at 2023-01-17 05:13:16.936988+00:00) securityworker stdout | 2023-01-17 05:13:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:46 UTC)" executed successfully builder stdout | 2023-01-17 05:13:17,772 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:17,785 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:17,785 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:17,796 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:13:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:13:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:04.936835+00:00 (in 45.590228 seconds) repositorygcworker stdout | 2023-01-17 05:13:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:19 UTC)" (scheduled at 2023-01-17 05:13:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:13:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:13:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:13:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:46.936988+00:00 (in 27.399055 seconds) securityworker stdout | 2023-01-17 05:13:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:49 UTC)" (scheduled at 2023-01-17 05:13:19.537472+00:00) securityworker stdout | 2023-01-17 05:13:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:13:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:13:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:13:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:13:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:13:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:13:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:13:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:13:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:51.348605+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 05:13:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:51 UTC)" (scheduled at 2023-01-17 05:13:21.348605+00:00) gcworker stdout | 2023-01-17 05:13:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722801349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:13:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:13:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:13:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:13:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:13:22,060 [236] [DEBUG] [app] Starting request: urn:request:f982291a-ef89-4de9-a5c7-ccf9110d5029 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:22,060 [231] [DEBUG] [app] Starting request: urn:request:b8be57a1-ab70-4e86-8652-e4429af7f1b2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:22,066 [243] [DEBUG] [app] Starting request: urn:request:cc5aa1b7-fd0f-4e8e-bf53-9029766581ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:22,066 [243] [DEBUG] [app] Ending request: urn:request:cc5aa1b7-fd0f-4e8e-bf53-9029766581ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc5aa1b7-fd0f-4e8e-bf53-9029766581ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:22,065 [239] [DEBUG] [app] Starting request: urn:request:e60e33d0-07af-45b9-971e-fe41742ef121 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:13:22,067 [239] [DEBUG] [app] Ending request: urn:request:e60e33d0-07af-45b9-971e-fe41742ef121 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e60e33d0-07af-45b9-971e-fe41742ef121', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:13:22,067 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.003 162 0.003) gunicorn-web stdout | 2023-01-17 05:13:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:22,071 [235] [DEBUG] [app] Starting request: urn:request:2a550fc9-51e9-46fc-a995-024944795118 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:22,071 [231] [DEBUG] [app] Starting request: urn:request:fc551f0b-d3da-4db7-9292-bcb8cceb427f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:22,072 [235] [DEBUG] [app] Ending request: urn:request:2a550fc9-51e9-46fc-a995-024944795118 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2a550fc9-51e9-46fc-a995-024944795118', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:22,072 [231] [DEBUG] [app] Ending request: urn:request:fc551f0b-d3da-4db7-9292-bcb8cceb427f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc551f0b-d3da-4db7-9292-bcb8cceb427f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:13:22,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:22,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:22,089 [231] [DEBUG] [app] Ending request: urn:request:b8be57a1-ab70-4e86-8652-e4429af7f1b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b8be57a1-ab70-4e86-8652-e4429af7f1b2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:22,089 [236] [DEBUG] [app] Ending request: urn:request:f982291a-ef89-4de9-a5c7-ccf9110d5029 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f982291a-ef89-4de9-a5c7-ccf9110d5029', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:13:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:13:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:35.646354+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 05:13:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:35 UTC)" (scheduled at 2023-01-17 05:13:25.646354+00:00) notificationworker stdout | 2023-01-17 05:13:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 25, 647286), True, datetime.datetime(2023, 1, 17, 5, 13, 25, 647286), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:35 UTC)" executed successfully builder stdout | 2023-01-17 05:13:27,801 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:27,813 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:27,813 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:27,824 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:27,875 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:13:27,935 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:13:28,865 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:13:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:13:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:35.853396+00:00 (in 6.889285 seconds) exportactionlogsworker stdout | 2023-01-17 05:13:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:28 UTC)" (scheduled at 2023-01-17 05:13:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:13:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:13:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:28 UTC)" executed successfully builder stdout | 2023-01-17 05:13:29,326 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:13:29,338 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:13:29,766 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:13:29,847 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:13:30,797 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:13:30,800 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:13:30,802 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:13:30,805 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:13:30,807 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:13:32,935 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:13:33,225 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:13:34,371 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:13:34,384 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:13:34,594 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:13:34,798 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:13:34,800 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:13:34,802 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:13:35,060 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:13:35,063 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:13:35,066 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:13:35,068 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:13:35,071 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:13:35,073 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:13:35,078 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:13:35,085 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:13:35,090 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:13:35,150 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:13:35,389 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:13:35,594 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} notificationworker stdout | 2023-01-17 05:13:35,647 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:36.635997+00:00 (in 0.988457 seconds) notificationworker stdout | 2023-01-17 05:13:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:45 UTC)" (scheduled at 2023-01-17 05:13:35.646354+00:00) notificationworker stdout | 2023-01-17 05:13:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 35, 647960), True, datetime.datetime(2023, 1, 17, 5, 13, 35, 647960), 0, 'notification/%', 50, 1, 0]) repositoryactioncounter stdout | 2023-01-17 05:13:35,652 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:13:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:35,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:35,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:13:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:13:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:28.963662+00:00 (in 53.109784 seconds) exportactionlogsworker stdout | 2023-01-17 05:13:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:35 UTC)" (scheduled at 2023-01-17 05:13:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:13:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:13:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 35, 854247), True, datetime.datetime(2023, 1, 17, 5, 13, 35, 854247), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:13:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:13:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:13:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:13:36,123 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:13:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:45.646354+00:00 (in 9.009905 seconds) notificationworker stdout | 2023-01-17 05:13:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:18:36 UTC)" (scheduled at 2023-01-17 05:13:36.635997+00:00) notificationworker stdout | 2023-01-17 05:13:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 13, 36, 636790), 'notification/%']) teamsyncworker stdout | 2023-01-17 05:13:36,638 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} notificationworker stdout | 2023-01-17 05:13:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 13, 36, 636790), True, datetime.datetime(2023, 1, 17, 5, 13, 36, 636790), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:13:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 13, 36, 636790), True, datetime.datetime(2023, 1, 17, 5, 13, 36, 636790), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 13, 36, 636790), 'notification/%']) notificationworker stdout | 2023-01-17 05:13:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:18:36 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:13:36,743 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:13:36,954 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:13:37,060 [236] [DEBUG] [app] Starting request: urn:request:22926385-216a-47a7-81be-8f92e4525733 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:37,060 [231] [DEBUG] [app] Starting request: urn:request:860fb5e3-7baa-4675-9d2b-28f150537986 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:37,065 [242] [DEBUG] [app] Starting request: urn:request:975225da-dc97-4b7a-bf9d-d083ff256a97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:37,066 [243] [DEBUG] [app] Starting request: urn:request:c9fe7585-c529-4bf5-ae2a-a63f9ea1be5c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:37,066 [242] [DEBUG] [app] Ending request: urn:request:975225da-dc97-4b7a-bf9d-d083ff256a97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:975225da-dc97-4b7a-bf9d-d083ff256a97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:13:37,066 [243] [DEBUG] [app] Ending request: urn:request:c9fe7585-c529-4bf5-ae2a-a63f9ea1be5c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c9fe7585-c529-4bf5-ae2a-a63f9ea1be5c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:13:37,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:13:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,069 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:37,071 [235] [DEBUG] [app] Starting request: urn:request:3e60a5b0-45da-45c4-8efa-c29e38b787b1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:37,071 [236] [DEBUG] [app] Starting request: urn:request:5cabb000-96d1-4e01-a3cb-27164f925d1f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:37,071 [235] [DEBUG] [app] Ending request: urn:request:3e60a5b0-45da-45c4-8efa-c29e38b787b1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e60a5b0-45da-45c4-8efa-c29e38b787b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:37,072 [236] [DEBUG] [app] Ending request: urn:request:5cabb000-96d1-4e01-a3cb-27164f925d1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5cabb000-96d1-4e01-a3cb-27164f925d1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:13:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:37,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:37,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:37,088 [236] [DEBUG] [app] Ending request: urn:request:22926385-216a-47a7-81be-8f92e4525733 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:22926385-216a-47a7-81be-8f92e4525733', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:13:37,088 [231] [DEBUG] [app] Ending request: urn:request:860fb5e3-7baa-4675-9d2b-28f150537986 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:860fb5e3-7baa-4675-9d2b-28f150537986', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:13:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:13:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:37.537407+00:00 (in 0.280340 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:13:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:37 UTC)" (scheduled at 2023-01-17 05:13:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:13:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:13:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 37, 257485), True, datetime.datetime(2023, 1, 17, 5, 13, 37, 257485), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:13:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:13:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:13:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:13:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:13:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:37.256652+00:00 (in 59.718847 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:13:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:37 UTC)" (scheduled at 2023-01-17 05:13:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:13:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:13:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:37 UTC)" executed successfully builder stdout | 2023-01-17 05:13:37,834 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:37,846 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:37,846 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:37,856 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:13:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:13:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:10.738568+00:00 (in 29.999533 seconds) buildlogsarchiver stdout | 2023-01-17 05:13:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:10 UTC)" (scheduled at 2023-01-17 05:13:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:13:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 13, 40, 739416), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:13:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:13:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:13:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:13:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:13:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:42.941536+00:00 (in 1.792771 seconds) namespacegcworker stdout | 2023-01-17 05:13:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:41 UTC)" (scheduled at 2023-01-17 05:13:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:13:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:13:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 41, 149172), True, datetime.datetime(2023, 1, 17, 5, 13, 41, 149172), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:13:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:13:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:13:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:13:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:13:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:41.148245+00:00 (in 58.206228 seconds) namespacegcworker stdout | 2023-01-17 05:13:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:42 UTC)" (scheduled at 2023-01-17 05:13:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:13:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:13:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:14:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:13:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:55.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:13:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:55 UTC)" (scheduled at 2023-01-17 05:13:45.646354+00:00) notificationworker stdout | 2023-01-17 05:13:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 45, 647211), True, datetime.datetime(2023, 1, 17, 5, 13, 45, 647211), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:13:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:13:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:13:49.537472+00:00 (in 2.600005 seconds) securityworker stdout | 2023-01-17 05:13:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:16 UTC)" (scheduled at 2023-01-17 05:13:46.936988+00:00) securityworker stdout | 2023-01-17 05:13:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:16 UTC)" executed successfully builder stdout | 2023-01-17 05:13:47,866 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:47,878 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:47,878 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:47,889 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:13:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:16.936988+00:00 (in 27.399019 seconds) securityworker stdout | 2023-01-17 05:13:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:19 UTC)" (scheduled at 2023-01-17 05:13:49.537472+00:00) securityworker stdout | 2023-01-17 05:13:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:13:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:13:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:13:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:13:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:13:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:13:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:13:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:13:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:13:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:13:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:13:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:13:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:13:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:13:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:13:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:21.348605+00:00 (in 29.999478 seconds) gcworker stdout | 2023-01-17 05:13:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:21 UTC)" (scheduled at 2023-01-17 05:13:51.348605+00:00) gcworker stdout | 2023-01-17 05:13:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722831349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:13:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:13:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:13:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:13:52,060 [231] [DEBUG] [app] Starting request: urn:request:402a01ed-bce8-4036-9fff-68d731372764 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:52,060 [235] [DEBUG] [app] Starting request: urn:request:1c2fc8a2-fa2f-4ade-92cb-538b80f65dfd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:13:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:13:52,065 [230] [DEBUG] [app] Starting request: urn:request:af429813-9e24-48f6-abbd-63f652c480bd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:52,065 [243] [DEBUG] [app] Starting request: urn:request:4e3b4520-5d16-493a-87f3-8906a27fbc64 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:13:52,065 [230] [DEBUG] [app] Ending request: urn:request:af429813-9e24-48f6-abbd-63f652c480bd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af429813-9e24-48f6-abbd-63f652c480bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:52,066 [243] [DEBUG] [app] Ending request: urn:request:4e3b4520-5d16-493a-87f3-8906a27fbc64 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e3b4520-5d16-493a-87f3-8906a27fbc64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:13:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:13:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:13:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:13:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:13:52,071 [236] [DEBUG] [app] Starting request: urn:request:df53880f-d5c6-45de-995f-c489b664c799 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:52,071 [235] [DEBUG] [app] Starting request: urn:request:b6580ac8-a9aa-422d-96d9-22704a4b304d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:13:52,071 [236] [DEBUG] [app] Ending request: urn:request:df53880f-d5c6-45de-995f-c489b664c799 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df53880f-d5c6-45de-995f-c489b664c799', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:52,071 [235] [DEBUG] [app] Ending request: urn:request:b6580ac8-a9aa-422d-96d9-22704a4b304d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6580ac8-a9aa-422d-96d9-22704a4b304d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:13:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:13:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:13:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:13:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:13:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:13:52,088 [235] [DEBUG] [app] Ending request: urn:request:1c2fc8a2-fa2f-4ade-92cb-538b80f65dfd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c2fc8a2-fa2f-4ade-92cb-538b80f65dfd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:52,088 [231] [DEBUG] [app] Ending request: urn:request:402a01ed-bce8-4036-9fff-68d731372764 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:402a01ed-bce8-4036-9fff-68d731372764', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:13:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:13:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:13:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) notificationworker stdout | 2023-01-17 05:13:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:13:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:01.636362+00:00 (in 5.989519 seconds) notificationworker stdout | 2023-01-17 05:13:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:05 UTC)" (scheduled at 2023-01-17 05:13:55.646354+00:00) notificationworker stdout | 2023-01-17 05:13:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:13:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 13, 55, 647228), True, datetime.datetime(2023, 1, 17, 5, 13, 55, 647228), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:13:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:13:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:13:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:13:57,890 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} builder stdout | 2023-01-17 05:13:57,899 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:13:57,911 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:13:57,911 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:13:57,921 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:13:57,948 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:13:58,890 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:13:59,362 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:13:59,373 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:13:59,781 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:13:59,882 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:14:00,804 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:14:00,807 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:14:00,809 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:14:00,811 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:14:00,814 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:14:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:05.646354+00:00 (in 4.009491 seconds) notificationworker stdout | 2023-01-17 05:14:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:01 UTC)" (scheduled at 2023-01-17 05:14:01.636362+00:00) notificationworker stdout | 2023-01-17 05:14:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:14:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:14:02,971 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:14:03,261 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:14:04,402 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:14:04,418 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:14:04,602 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:14:04,804 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:14:04,806 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:14:04,809 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:14:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:14:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:19.346120+00:00 (in 14.408864 seconds) repositorygcworker stdout | 2023-01-17 05:14:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:04 UTC)" (scheduled at 2023-01-17 05:14:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:14:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:14:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 4, 937699), True, datetime.datetime(2023, 1, 17, 5, 14, 4, 937699), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:14:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:14:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:14:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:14:05,068 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:14:05,070 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:14:05,076 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:14:05,078 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:14:05,081 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:14:05,083 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:14:05,088 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:14:05,094 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:14:05,098 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:14:05,160 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:14:05,400 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:14:05,600 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} notificationworker stdout | 2023-01-17 05:14:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:15.646354+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:14:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:15 UTC)" (scheduled at 2023-01-17 05:14:05.646354+00:00) notificationworker stdout | 2023-01-17 05:14:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 5, 647225), True, datetime.datetime(2023, 1, 17, 5, 14, 5, 647225), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:15 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:14:05,687 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:14:06,150 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:14:06,652 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:14:06,764 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:14:06,966 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:14:07,060 [235] [DEBUG] [app] Starting request: urn:request:0e03c240-5cc7-4fe6-aa24-dfa91bcabbfd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:07,060 [236] [DEBUG] [app] Starting request: urn:request:ae7d02a3-caed-46ad-beda-1ebebedaf000 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:07,065 [230] [DEBUG] [app] Starting request: urn:request:cf73a2d6-4999-4df8-82d0-9ce5b5910040 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:07,065 [243] [DEBUG] [app] Starting request: urn:request:0cb801eb-615a-411d-9d92-a7bdda4e7cf7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:07,066 [230] [DEBUG] [app] Ending request: urn:request:cf73a2d6-4999-4df8-82d0-9ce5b5910040 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf73a2d6-4999-4df8-82d0-9ce5b5910040', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:07,066 [243] [DEBUG] [app] Ending request: urn:request:0cb801eb-615a-411d-9d92-a7bdda4e7cf7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0cb801eb-615a-411d-9d92-a7bdda4e7cf7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:14:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:14:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:07,071 [231] [DEBUG] [app] Starting request: urn:request:fccca104-a824-4870-9698-e6d3fd04a8c3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:07,071 [236] [DEBUG] [app] Starting request: urn:request:ccbaee7e-0d1e-47fe-abcd-45e9a466c350 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:07,071 [236] [DEBUG] [app] Ending request: urn:request:ccbaee7e-0d1e-47fe-abcd-45e9a466c350 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ccbaee7e-0d1e-47fe-abcd-45e9a466c350', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:07,071 [231] [DEBUG] [app] Ending request: urn:request:fccca104-a824-4870-9698-e6d3fd04a8c3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fccca104-a824-4870-9698-e6d3fd04a8c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:14:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:07,088 [236] [DEBUG] [app] Ending request: urn:request:ae7d02a3-caed-46ad-beda-1ebebedaf000 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ae7d02a3-caed-46ad-beda-1ebebedaf000', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:07,088 [235] [DEBUG] [app] Ending request: urn:request:0e03c240-5cc7-4fe6-aa24-dfa91bcabbfd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e03c240-5cc7-4fe6-aa24-dfa91bcabbfd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:14:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:14:07,922 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:07,934 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:07,934 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:07,945 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:14:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:14:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:40.738568+00:00 (in 29.999565 seconds) buildlogsarchiver stdout | 2023-01-17 05:14:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:40 UTC)" (scheduled at 2023-01-17 05:14:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:14:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 14, 10, 739448), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:14:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:14:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:14:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:14:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:25.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:14:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:25 UTC)" (scheduled at 2023-01-17 05:14:15.646354+00:00) notificationworker stdout | 2023-01-17 05:14:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 15, 647175), True, datetime.datetime(2023, 1, 17, 5, 14, 15, 647175), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:14:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:19.537472+00:00 (in 2.599992 seconds) securityworker stdout | 2023-01-17 05:14:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:46 UTC)" (scheduled at 2023-01-17 05:14:16.936988+00:00) securityworker stdout | 2023-01-17 05:14:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:46 UTC)" executed successfully builder stdout | 2023-01-17 05:14:17,955 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:17,968 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:17,968 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:17,979 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:14:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:14:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:04.936835+00:00 (in 45.590265 seconds) repositorygcworker stdout | 2023-01-17 05:14:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:19 UTC)" (scheduled at 2023-01-17 05:14:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:14:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:14:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:14:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:46.936988+00:00 (in 27.399053 seconds) securityworker stdout | 2023-01-17 05:14:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:49 UTC)" (scheduled at 2023-01-17 05:14:19.537472+00:00) securityworker stdout | 2023-01-17 05:14:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:14:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:14:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:14:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:14:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:14:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:14:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:14:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:14:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:51.348605+00:00 (in 29.999492 seconds) gcworker stdout | 2023-01-17 05:14:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:51 UTC)" (scheduled at 2023-01-17 05:14:21.348605+00:00) gcworker stdout | 2023-01-17 05:14:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722861349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:14:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:14:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:14:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:14:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:14:22,059 [231] [DEBUG] [app] Starting request: urn:request:077f7613-33b9-4838-8c07-b701a3fb101f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:22,059 [233] [DEBUG] [app] Starting request: urn:request:84a67f5d-d50d-4b8b-acfd-58122f8f4487 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:22,064 [230] [DEBUG] [app] Starting request: urn:request:c26c74f1-7f72-4b5b-961c-513139401c1d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:22,065 [243] [DEBUG] [app] Starting request: urn:request:10a6e6cd-0e58-4d1c-8b80-89b0dc7d5881 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:22,065 [230] [DEBUG] [app] Ending request: urn:request:c26c74f1-7f72-4b5b-961c-513139401c1d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c26c74f1-7f72-4b5b-961c-513139401c1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:14:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:14:22,065 [243] [DEBUG] [app] Ending request: urn:request:10a6e6cd-0e58-4d1c-8b80-89b0dc7d5881 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:10a6e6cd-0e58-4d1c-8b80-89b0dc7d5881', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:22,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:14:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:22,069 [231] [DEBUG] [app] Starting request: urn:request:b0e1d25b-f94c-417e-bd21-0eca67fd2c3f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:22,070 [231] [DEBUG] [app] Ending request: urn:request:b0e1d25b-f94c-417e-bd21-0eca67fd2c3f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b0e1d25b-f94c-417e-bd21-0eca67fd2c3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:14:22,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:22,070 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:22,071 [236] [DEBUG] [app] Starting request: urn:request:7d5fbae3-a6d0-4696-ad57-b1bba0ef7be9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:22,072 [236] [DEBUG] [app] Ending request: urn:request:7d5fbae3-a6d0-4696-ad57-b1bba0ef7be9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d5fbae3-a6d0-4696-ad57-b1bba0ef7be9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:14:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:22,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:22,087 [233] [DEBUG] [app] Ending request: urn:request:84a67f5d-d50d-4b8b-acfd-58122f8f4487 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84a67f5d-d50d-4b8b-acfd-58122f8f4487', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:22,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:14:22,088 [231] [DEBUG] [app] Ending request: urn:request:077f7613-33b9-4838-8c07-b701a3fb101f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:077f7613-33b9-4838-8c07-b701a3fb101f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:22,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:14:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:14:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:35.646354+00:00 (in 9.999582 seconds) notificationworker stdout | 2023-01-17 05:14:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:35 UTC)" (scheduled at 2023-01-17 05:14:25.646354+00:00) notificationworker stdout | 2023-01-17 05:14:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 25, 647168), True, datetime.datetime(2023, 1, 17, 5, 14, 25, 647168), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:14:27,925 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:14:27,958 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:14:27,989 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:28,001 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:28,001 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:28,012 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:14:28,926 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} exportactionlogsworker stdout | 2023-01-17 05:14:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:14:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:35.853396+00:00 (in 6.889283 seconds) exportactionlogsworker stdout | 2023-01-17 05:14:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:28 UTC)" (scheduled at 2023-01-17 05:14:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:14:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:14:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:28 UTC)" executed successfully builder stdout | 2023-01-17 05:14:29,368 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:14:29,378 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:14:29,790 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:14:29,917 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:14:30,812 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:14:30,814 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:14:30,817 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:14:30,820 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:14:30,822 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:14:32,978 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:14:33,296 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:14:34,431 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:14:34,436 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:14:34,632 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:14:34,811 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:14:34,814 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:14:34,816 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:14:35,074 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:14:35,077 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:14:35,086 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:14:35,089 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:14:35,092 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:14:35,094 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:14:35,097 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:14:35,103 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:14:35,105 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:14:35,166 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:14:35,435 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:14:35,622 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} notificationworker stdout | 2023-01-17 05:14:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:45.646354+00:00 (in 9.999583 seconds) notificationworker stdout | 2023-01-17 05:14:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:45 UTC)" (scheduled at 2023-01-17 05:14:35.646354+00:00) notificationworker stdout | 2023-01-17 05:14:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 35, 647131), True, datetime.datetime(2023, 1, 17, 5, 14, 35, 647131), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:45 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:14:35,713 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} exportactionlogsworker stdout | 2023-01-17 05:14:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:14:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:28.963662+00:00 (in 53.109772 seconds) exportactionlogsworker stdout | 2023-01-17 05:14:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:35 UTC)" (scheduled at 2023-01-17 05:14:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:14:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:14:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 35, 854270), True, datetime.datetime(2023, 1, 17, 5, 14, 35, 854270), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:14:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:14:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:14:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:14:36,183 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:14:36,688 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:14:36,799 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:14:37,001 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:14:37,060 [236] [DEBUG] [app] Starting request: urn:request:6d3cfafb-3b1f-461c-bd8e-81d47f9ea750 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:37,060 [235] [DEBUG] [app] Starting request: urn:request:e6bf6356-0393-44a0-955c-ef81b837bf0d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:37,065 [230] [DEBUG] [app] Starting request: urn:request:4044ec25-8ada-4eab-b4e4-8ce2a56f2484 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:37,065 [243] [DEBUG] [app] Starting request: urn:request:127e8c6b-6cf1-48c2-8293-1a895b181fde (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:37,065 [230] [DEBUG] [app] Ending request: urn:request:4044ec25-8ada-4eab-b4e4-8ce2a56f2484 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4044ec25-8ada-4eab-b4e4-8ce2a56f2484', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:14:37,066 [243] [DEBUG] [app] Ending request: urn:request:127e8c6b-6cf1-48c2-8293-1a895b181fde (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:127e8c6b-6cf1-48c2-8293-1a895b181fde', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:14:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:14:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:37,070 [231] [DEBUG] [app] Starting request: urn:request:b3b53481-50aa-4964-8384-87b936b62bc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:37,071 [231] [DEBUG] [app] Ending request: urn:request:b3b53481-50aa-4964-8384-87b936b62bc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3b53481-50aa-4964-8384-87b936b62bc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:37,071 [236] [DEBUG] [app] Starting request: urn:request:6211348f-169e-42de-a05a-9676ad4cf52e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:14:37,071 [236] [DEBUG] [app] Ending request: urn:request:6211348f-169e-42de-a05a-9676ad4cf52e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6211348f-169e-42de-a05a-9676ad4cf52e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:14:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:37,084 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:37,088 [236] [DEBUG] [app] Ending request: urn:request:6d3cfafb-3b1f-461c-bd8e-81d47f9ea750 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d3cfafb-3b1f-461c-bd8e-81d47f9ea750', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:14:37,093 [235] [DEBUG] [app] Ending request: urn:request:e6bf6356-0393-44a0-955c-ef81b837bf0d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6bf6356-0393-44a0-955c-ef81b837bf0d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:37,094 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:37,094 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) securityscanningnotificationworker stdout | 2023-01-17 05:14:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:14:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:37.537407+00:00 (in 0.280345 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:14:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:37 UTC)" (scheduled at 2023-01-17 05:14:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:14:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:14:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 37, 257469), True, datetime.datetime(2023, 1, 17, 5, 14, 37, 257469), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:14:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:14:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:14:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:14:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:14:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:37.256652+00:00 (in 59.718856 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:14:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:37 UTC)" (scheduled at 2023-01-17 05:14:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:14:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:14:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:37 UTC)" executed successfully builder stdout | 2023-01-17 05:14:38,022 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:38,034 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:38,034 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:38,044 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:14:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:14:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:10.738568+00:00 (in 29.999567 seconds) buildlogsarchiver stdout | 2023-01-17 05:14:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:10 UTC)" (scheduled at 2023-01-17 05:14:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:14:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 14, 40, 739404), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:14:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:14:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:14:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:14:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:14:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:42.941536+00:00 (in 1.792834 seconds) namespacegcworker stdout | 2023-01-17 05:14:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:41 UTC)" (scheduled at 2023-01-17 05:14:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:14:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:14:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 41, 149125), True, datetime.datetime(2023, 1, 17, 5, 14, 41, 149125), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:14:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:14:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:14:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:14:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:14:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:41.148245+00:00 (in 58.206214 seconds) namespacegcworker stdout | 2023-01-17 05:14:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:42 UTC)" (scheduled at 2023-01-17 05:14:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:14:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:14:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:15:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:14:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:55.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:14:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:55 UTC)" (scheduled at 2023-01-17 05:14:45.646354+00:00) notificationworker stdout | 2023-01-17 05:14:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 45, 647231), True, datetime.datetime(2023, 1, 17, 5, 14, 45, 647231), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:14:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:14:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:14:49.537472+00:00 (in 2.600037 seconds) securityworker stdout | 2023-01-17 05:14:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:16 UTC)" (scheduled at 2023-01-17 05:14:46.936988+00:00) securityworker stdout | 2023-01-17 05:14:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:16 UTC)" executed successfully builder stdout | 2023-01-17 05:14:48,055 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:48,067 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:48,067 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:48,078 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:14:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:16.936988+00:00 (in 27.399090 seconds) securityworker stdout | 2023-01-17 05:14:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:19 UTC)" (scheduled at 2023-01-17 05:14:49.537472+00:00) securityworker stdout | 2023-01-17 05:14:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:14:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:14:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:14:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:14:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:14:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:14:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:14:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:14:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:14:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:14:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:14:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:14:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:14:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:14:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:14:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:21.348605+00:00 (in 29.999482 seconds) gcworker stdout | 2023-01-17 05:14:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:21 UTC)" (scheduled at 2023-01-17 05:14:51.348605+00:00) gcworker stdout | 2023-01-17 05:14:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722891349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:14:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:14:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:14:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:14:52,059 [236] [DEBUG] [app] Starting request: urn:request:cc73ac77-6146-4850-8335-8b88bae0ebad (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:52,060 [233] [DEBUG] [app] Starting request: urn:request:471c51e9-d5cc-4393-a39e-55cab560e9c9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:14:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:14:52,064 [230] [DEBUG] [app] Starting request: urn:request:4de1c520-aa7f-450c-a20d-723c7853db04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:52,065 [242] [DEBUG] [app] Starting request: urn:request:38d058d8-ef02-4477-98c2-48604e555f23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:14:52,065 [230] [DEBUG] [app] Ending request: urn:request:4de1c520-aa7f-450c-a20d-723c7853db04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4de1c520-aa7f-450c-a20d-723c7853db04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:14:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:14:52,065 [242] [DEBUG] [app] Ending request: urn:request:38d058d8-ef02-4477-98c2-48604e555f23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:38d058d8-ef02-4477-98c2-48604e555f23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:14:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:14:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:14:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:14:52,070 [231] [DEBUG] [app] Starting request: urn:request:55a5552d-a118-4a48-a2d1-7e3f2c584ff7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:52,070 [236] [DEBUG] [app] Starting request: urn:request:af793fc9-fe1f-497e-9282-cc0af93743bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:14:52,071 [236] [DEBUG] [app] Ending request: urn:request:af793fc9-fe1f-497e-9282-cc0af93743bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af793fc9-fe1f-497e-9282-cc0af93743bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:52,071 [231] [DEBUG] [app] Ending request: urn:request:55a5552d-a118-4a48-a2d1-7e3f2c584ff7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55a5552d-a118-4a48-a2d1-7e3f2c584ff7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:14:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:14:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:14:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:14:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:14:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:52,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:14:52,087 [236] [DEBUG] [app] Ending request: urn:request:cc73ac77-6146-4850-8335-8b88bae0ebad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc73ac77-6146-4850-8335-8b88bae0ebad', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:14:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:14:52,088 [233] [DEBUG] [app] Ending request: urn:request:471c51e9-d5cc-4393-a39e-55cab560e9c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:471c51e9-d5cc-4393-a39e-55cab560e9c9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:14:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:14:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 05:14:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:14:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:01.636362+00:00 (in 5.989528 seconds) notificationworker stdout | 2023-01-17 05:14:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:05 UTC)" (scheduled at 2023-01-17 05:14:55.646354+00:00) notificationworker stdout | 2023-01-17 05:14:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:14:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 14, 55, 647261), True, datetime.datetime(2023, 1, 17, 5, 14, 55, 647261), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:14:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:14:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:14:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:14:57,946 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:14:57,977 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:14:58,088 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:14:58,100 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:14:58,100 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:14:58,111 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:14:58,941 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:14:59,389 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:14:59,402 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:14:59,824 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:14:59,926 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:15:00,819 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:15:00,822 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:15:00,824 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:15:00,826 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:15:00,829 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:15:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:05.646354+00:00 (in 4.009542 seconds) notificationworker stdout | 2023-01-17 05:15:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:01 UTC)" (scheduled at 2023-01-17 05:15:01.636362+00:00) notificationworker stdout | 2023-01-17 05:15:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:15:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:15:02,998 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:15:03,309 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:15:04,467 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:15:04,471 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:15:04,661 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:15:04,818 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:15:04,820 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:15:04,822 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:15:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:15:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:19.346120+00:00 (in 14.408883 seconds) repositorygcworker stdout | 2023-01-17 05:15:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:04 UTC)" (scheduled at 2023-01-17 05:15:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:15:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:15:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 4, 937669), True, datetime.datetime(2023, 1, 17, 5, 15, 4, 937669), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:15:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:15:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:15:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:15:05,084 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:15:05,087 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:15:05,095 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:15:05,098 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:15:05,101 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:15:05,103 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:15:05,105 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:15:05,111 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:15:05,113 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:15:05,201 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:15:05,446 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:15:05,639 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} notificationworker stdout | 2023-01-17 05:15:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:15.646354+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 05:15:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:15 UTC)" (scheduled at 2023-01-17 05:15:05.646354+00:00) notificationworker stdout | 2023-01-17 05:15:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 5, 647190), True, datetime.datetime(2023, 1, 17, 5, 15, 5, 647190), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:15 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:15:05,729 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:15:06,193 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:15:06,723 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:15:06,835 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:15:07,036 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:15:07,060 [231] [DEBUG] [app] Starting request: urn:request:527a9388-9745-49e7-a7a9-e45222ab13ec (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:07,060 [235] [DEBUG] [app] Starting request: urn:request:ff834aa0-62fd-435e-adb7-9cc589cb7326 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:07,065 [243] [DEBUG] [app] Starting request: urn:request:b1bffc68-a29b-4ea5-aab4-653ea1a890d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:07,065 [230] [DEBUG] [app] Starting request: urn:request:917724c1-a09f-443d-8c56-bcfe51610744 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:07,065 [243] [DEBUG] [app] Ending request: urn:request:b1bffc68-a29b-4ea5-aab4-653ea1a890d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1bffc68-a29b-4ea5-aab4-653ea1a890d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:15:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:15:07,066 [230] [DEBUG] [app] Ending request: urn:request:917724c1-a09f-443d-8c56-bcfe51610744 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:917724c1-a09f-443d-8c56-bcfe51610744', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:15:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:15:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:07,070 [233] [DEBUG] [app] Starting request: urn:request:0652d0ac-79f1-4afa-9b94-8ba16a5c090b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:07,071 [233] [DEBUG] [app] Ending request: urn:request:0652d0ac-79f1-4afa-9b94-8ba16a5c090b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0652d0ac-79f1-4afa-9b94-8ba16a5c090b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:07,071 [236] [DEBUG] [app] Starting request: urn:request:dede1cae-6a08-48cf-8fa0-c03dddda6bb5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:15:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:07,071 [236] [DEBUG] [app] Ending request: urn:request:dede1cae-6a08-48cf-8fa0-c03dddda6bb5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dede1cae-6a08-48cf-8fa0-c03dddda6bb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:07,088 [231] [DEBUG] [app] Ending request: urn:request:527a9388-9745-49e7-a7a9-e45222ab13ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:527a9388-9745-49e7-a7a9-e45222ab13ec', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:07,088 [235] [DEBUG] [app] Ending request: urn:request:ff834aa0-62fd-435e-adb7-9cc589cb7326 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ff834aa0-62fd-435e-adb7-9cc589cb7326', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:07,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:15:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 05:15:08,121 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:08,133 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:08,133 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:08,144 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:15:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:15:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:40.738568+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2023-01-17 05:15:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:40 UTC)" (scheduled at 2023-01-17 05:15:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:15:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 15, 10, 739468), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:15:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:15:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:15:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:15:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:25.646354+00:00 (in 9.999579 seconds) notificationworker stdout | 2023-01-17 05:15:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:25 UTC)" (scheduled at 2023-01-17 05:15:15.646354+00:00) notificationworker stdout | 2023-01-17 05:15:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 15, 647185), True, datetime.datetime(2023, 1, 17, 5, 15, 15, 647185), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:15:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:19.537472+00:00 (in 2.599985 seconds) securityworker stdout | 2023-01-17 05:15:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:46 UTC)" (scheduled at 2023-01-17 05:15:16.936988+00:00) securityworker stdout | 2023-01-17 05:15:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:46 UTC)" executed successfully builder stdout | 2023-01-17 05:15:18,154 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:18,167 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:18,167 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:18,178 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:15:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:15:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:04.936835+00:00 (in 45.590230 seconds) repositorygcworker stdout | 2023-01-17 05:15:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:19 UTC)" (scheduled at 2023-01-17 05:15:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:15:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:15:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:15:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:46.936988+00:00 (in 27.399021 seconds) securityworker stdout | 2023-01-17 05:15:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:49 UTC)" (scheduled at 2023-01-17 05:15:19.537472+00:00) securityworker stdout | 2023-01-17 05:15:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:15:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:15:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:15:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:15:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:15:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:15:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:15:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:15:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:51.348605+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 05:15:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:51 UTC)" (scheduled at 2023-01-17 05:15:21.348605+00:00) gcworker stdout | 2023-01-17 05:15:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722921349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:15:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:15:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:15:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:15:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:15:22,060 [231] [DEBUG] [app] Starting request: urn:request:c7183e35-4ffc-41a2-8023-55cb6375cc24 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:22,060 [236] [DEBUG] [app] Starting request: urn:request:8f867e31-e167-46b6-8d32-54c59a006873 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:22,065 [230] [DEBUG] [app] Starting request: urn:request:909cf968-7b46-454a-8d56-5fb14849038e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:22,065 [243] [DEBUG] [app] Starting request: urn:request:1144ce51-0c51-4ae3-bac2-82cbaddd8091 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:22,066 [230] [DEBUG] [app] Ending request: urn:request:909cf968-7b46-454a-8d56-5fb14849038e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:909cf968-7b46-454a-8d56-5fb14849038e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:22,066 [243] [DEBUG] [app] Ending request: urn:request:1144ce51-0c51-4ae3-bac2-82cbaddd8091 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1144ce51-0c51-4ae3-bac2-82cbaddd8091', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:15:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:15:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:22,070 [233] [DEBUG] [app] Starting request: urn:request:e0f827c7-4d78-4689-8f82-44438c766749 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:22,070 [233] [DEBUG] [app] Ending request: urn:request:e0f827c7-4d78-4689-8f82-44438c766749 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0f827c7-4d78-4689-8f82-44438c766749', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:22,071 [236] [DEBUG] [app] Starting request: urn:request:1d303f99-4a5d-4a76-b80f-709ff964afde (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:22,071 [236] [DEBUG] [app] Ending request: urn:request:1d303f99-4a5d-4a76-b80f-709ff964afde (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d303f99-4a5d-4a76-b80f-709ff964afde', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:22,088 [236] [DEBUG] [app] Ending request: urn:request:8f867e31-e167-46b6-8d32-54c59a006873 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f867e31-e167-46b6-8d32-54c59a006873', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:22,088 [231] [DEBUG] [app] Ending request: urn:request:c7183e35-4ffc-41a2-8023-55cb6375cc24 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c7183e35-4ffc-41a2-8023-55cb6375cc24', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:15:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:15:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:35.646354+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:15:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:35 UTC)" (scheduled at 2023-01-17 05:15:25.646354+00:00) notificationworker stdout | 2023-01-17 05:15:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 25, 647248), True, datetime.datetime(2023, 1, 17, 5, 15, 25, 647248), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:15:27,982 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:15:27,986 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:15:28,188 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:28,201 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:28,201 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:28,211 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:15:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:15:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:35.853396+00:00 (in 6.889264 seconds) exportactionlogsworker stdout | 2023-01-17 05:15:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:28 UTC)" (scheduled at 2023-01-17 05:15:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:15:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:15:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:15:28,977 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:15:29,426 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:15:29,437 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:15:29,843 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:15:29,961 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} servicekey stdout | 2023-01-17 05:15:30,441 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 05:15:30,442 [88] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 05:15:30 UTC)" (scheduled at 2023-01-17 05:15:30.441589+00:00) servicekey stdout | 2023-01-17 05:15:30,442 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:30.441589+00:00 (in 3299.999643 seconds) servicekey stdout | 2023-01-17 05:15:30,442 [88] [DEBUG] [__main__] Starting automatic refresh of service key HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw to new expiration 2023-01-17 07:15:30.442488 servicekey stdout | 2023-01-17 05:15:30,443 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw', datetime.datetime(2023, 1, 10, 5, 15, 30, 443050), None, 1, 0]) servicekey stdout | 2023-01-17 05:15:30,454 [88] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443', 'HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw', 'quay', '{"n": "vyADstWkSWTJPDl7kqlPmO1eZbt9FNsmZ9_-tbg63McWVJDNdbRHLiI9N4_ZE8_B88DplENLF1WuADJhqVorEQVmjdVIBzD4kdVe4iOvR2mSA2eHlmDWzF5viMYFkKU_w2mHvCckDtnOpJaGnE-hozsrw625V5rDvMrC481O-6vcYOwDb5g3pk-lMMI2aAKURo-kS5osmLh73vHiGKqLN2ymMMV0zZHsa1jXfBLBMpRwtApJgBqdgA_87e3QPLGlOzwGv24FVCLECm-L_XxSQAGndNVnVCYS43gxZvjhqML1fPeCA3dD91GmGYsuwW7MMlYcvefd_8JwHXJO9eadlw", "e": "AQAB", "kty": "RSA", "kid": "HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw"}', '{"created_by": "CLI tool"}', datetime.datetime(2023, 1, 17, 4, 49, 11, 724988), datetime.datetime(2023, 1, 17, 7, 15, 30, 442488), None, 5, 5]) servicekey stdout | 2023-01-17 05:15:30,463 [88] [DEBUG] [__main__] Finished automatic refresh of service key HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw with new expiration 2023-01-17 07:15:30.442488 servicekey stdout | 2023-01-17 05:15:30,463 [88] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 05:15:30,463 [88] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 06:10:30 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:15:30,826 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:15:30,829 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:15:30,831 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:15:30,833 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:15:30,836 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:15:33,023 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:15:33,319 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:15:34,479 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:15:34,485 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:15:34,696 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:15:34,825 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:15:34,827 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:15:34,830 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:15:35,091 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:15:35,096 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:15:35,104 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:15:35,108 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:15:35,110 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:15:35,112 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:15:35,115 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:15:35,119 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:15:35,121 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:15:35,236 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:15:35,464 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:15:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:45.646354+00:00 (in 9.999504 seconds) notificationworker stdout | 2023-01-17 05:15:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:45 UTC)" (scheduled at 2023-01-17 05:15:35.646354+00:00) notificationworker stdout | 2023-01-17 05:15:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 35, 647324), True, datetime.datetime(2023, 1, 17, 5, 15, 35, 647324), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:45 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:15:35,673 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:15:35,744 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} exportactionlogsworker stdout | 2023-01-17 05:15:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:15:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:28.963662+00:00 (in 53.109739 seconds) exportactionlogsworker stdout | 2023-01-17 05:15:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:35 UTC)" (scheduled at 2023-01-17 05:15:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:15:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:15:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 35, 854375), True, datetime.datetime(2023, 1, 17, 5, 15, 35, 854375), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:15:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:15:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:15:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:15:36,228 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:15:36,759 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:15:36,869 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:15:37,054 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:15:37,059 [236] [DEBUG] [app] Starting request: urn:request:1daef30a-987c-4832-b31f-895635cd1783 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:37,059 [233] [DEBUG] [app] Starting request: urn:request:e633886f-3cc4-40d7-bef6-62c7e3a18d93 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:37,060 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:37,064 [230] [DEBUG] [app] Starting request: urn:request:317d0535-923d-49b2-82dd-ff5bc376939a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:37,064 [242] [DEBUG] [app] Starting request: urn:request:0cc9c230-a056-4bee-9c98-18b6637d4889 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:37,064 [230] [DEBUG] [app] Ending request: urn:request:317d0535-923d-49b2-82dd-ff5bc376939a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:317d0535-923d-49b2-82dd-ff5bc376939a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:15:37,065 [242] [DEBUG] [app] Ending request: urn:request:0cc9c230-a056-4bee-9c98-18b6637d4889 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0cc9c230-a056-4bee-9c98-18b6637d4889', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:15:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:37,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:37,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:37,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:37,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:37,070 [231] [DEBUG] [app] Starting request: urn:request:96e0ee2e-d847-4c19-8cb3-a25be57ec1c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:37,070 [236] [DEBUG] [app] Starting request: urn:request:83a0e534-84ee-4cac-804c-be505038e455 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:37,070 [231] [DEBUG] [app] Ending request: urn:request:96e0ee2e-d847-4c19-8cb3-a25be57ec1c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:96e0ee2e-d847-4c19-8cb3-a25be57ec1c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:37,070 [236] [DEBUG] [app] Ending request: urn:request:83a0e534-84ee-4cac-804c-be505038e455 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:83a0e534-84ee-4cac-804c-be505038e455', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:37,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:37,085 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:37,087 [236] [DEBUG] [app] Ending request: urn:request:1daef30a-987c-4832-b31f-895635cd1783 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1daef30a-987c-4832-b31f-895635cd1783', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:15:37,094 [233] [DEBUG] [app] Ending request: urn:request:e633886f-3cc4-40d7-bef6-62c7e3a18d93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e633886f-3cc4-40d7-bef6-62c7e3a18d93', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:37,094 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:37,094 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) securityscanningnotificationworker stdout | 2023-01-17 05:15:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:15:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:37.537407+00:00 (in 0.280371 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:15:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:37 UTC)" (scheduled at 2023-01-17 05:15:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:15:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:15:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 37, 257459), True, datetime.datetime(2023, 1, 17, 5, 15, 37, 257459), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:15:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:15:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:15:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:15:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:15:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:23.537048+00:00 (in 45.999234 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:15:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:37 UTC)" (scheduled at 2023-01-17 05:15:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:15:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:15:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:37 UTC)" executed successfully builder stdout | 2023-01-17 05:15:38,222 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:38,234 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:38,234 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:38,245 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:15:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:15:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:10.738568+00:00 (in 29.999519 seconds) buildlogsarchiver stdout | 2023-01-17 05:15:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:10 UTC)" (scheduled at 2023-01-17 05:15:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:15:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 15, 40, 739427), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:15:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:15:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:15:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:15:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:15:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:42.941536+00:00 (in 1.792809 seconds) namespacegcworker stdout | 2023-01-17 05:15:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:41 UTC)" (scheduled at 2023-01-17 05:15:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:15:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:15:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 41, 149171), True, datetime.datetime(2023, 1, 17, 5, 15, 41, 149171), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:15:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:15:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:15:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:15:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:15:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:41.148245+00:00 (in 58.206221 seconds) namespacegcworker stdout | 2023-01-17 05:15:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:42 UTC)" (scheduled at 2023-01-17 05:15:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:15:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:15:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:16:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:15:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:55.646354+00:00 (in 9.999496 seconds) notificationworker stdout | 2023-01-17 05:15:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:55 UTC)" (scheduled at 2023-01-17 05:15:45.646354+00:00) notificationworker stdout | 2023-01-17 05:15:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 45, 647254), True, datetime.datetime(2023, 1, 17, 5, 15, 45, 647254), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:15:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:15:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:15:49.537472+00:00 (in 2.600024 seconds) securityworker stdout | 2023-01-17 05:15:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:16 UTC)" (scheduled at 2023-01-17 05:15:46.936988+00:00) securityworker stdout | 2023-01-17 05:15:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:16 UTC)" executed successfully builder stdout | 2023-01-17 05:15:48,255 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:48,268 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:48,268 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:48,279 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:15:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:16.936988+00:00 (in 27.399063 seconds) securityworker stdout | 2023-01-17 05:15:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:19 UTC)" (scheduled at 2023-01-17 05:15:49.537472+00:00) securityworker stdout | 2023-01-17 05:15:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:15:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:15:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:15:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:15:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:15:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:15:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:15:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:15:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:15:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:15:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:15:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:15:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:15:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:15:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:15:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:21.348605+00:00 (in 29.999509 seconds) gcworker stdout | 2023-01-17 05:15:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:21 UTC)" (scheduled at 2023-01-17 05:15:51.348605+00:00) gcworker stdout | 2023-01-17 05:15:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722951349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:15:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:15:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:15:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:15:52,060 [233] [DEBUG] [app] Starting request: urn:request:c4e7308e-f64d-4e6c-bf43-377d14963fe9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:52,060 [231] [DEBUG] [app] Starting request: urn:request:260c4d98-177f-446a-9eb1-4ade62b8b975 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:15:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:15:52,065 [243] [DEBUG] [app] Starting request: urn:request:d96fa7ce-7f0b-44e5-abcc-555e2a8e218e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,065 [237] [DEBUG] [app] Starting request: urn:request:6e6f83f4-de92-418b-9fb2-580c34d15883 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,066 [243] [DEBUG] [app] Ending request: urn:request:d96fa7ce-7f0b-44e5-abcc-555e2a8e218e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d96fa7ce-7f0b-44e5-abcc-555e2a8e218e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,066 [237] [DEBUG] [app] Ending request: urn:request:6e6f83f4-de92-418b-9fb2-580c34d15883 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e6f83f4-de92-418b-9fb2-580c34d15883', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:15:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:15:52,066 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:15:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:15:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:15:52,071 [236] [DEBUG] [app] Starting request: urn:request:48740775-f125-4d38-90d5-e5754868e517 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:52,071 [235] [DEBUG] [app] Starting request: urn:request:95e3efe7-b605-49ec-a269-928196794975 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:15:52,071 [236] [DEBUG] [app] Ending request: urn:request:48740775-f125-4d38-90d5-e5754868e517 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48740775-f125-4d38-90d5-e5754868e517', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:15:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:52,071 [235] [DEBUG] [app] Ending request: urn:request:95e3efe7-b605-49ec-a269-928196794975 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95e3efe7-b605-49ec-a269-928196794975', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:52,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:15:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:15:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:15:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:15:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:52,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:15:52,088 [231] [DEBUG] [app] Ending request: urn:request:260c4d98-177f-446a-9eb1-4ade62b8b975 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:260c4d98-177f-446a-9eb1-4ade62b8b975', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:15:52,088 [233] [DEBUG] [app] Ending request: urn:request:c4e7308e-f64d-4e6c-bf43-377d14963fe9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4e7308e-f64d-4e6c-bf43-377d14963fe9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:15:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:15:52,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:15:52,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:15:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:15:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:01.636362+00:00 (in 5.989511 seconds) notificationworker stdout | 2023-01-17 05:15:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:05 UTC)" (scheduled at 2023-01-17 05:15:55.646354+00:00) notificationworker stdout | 2023-01-17 05:15:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:15:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 15, 55, 647252), True, datetime.datetime(2023, 1, 17, 5, 15, 55, 647252), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:15:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:15:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:15:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:15:57,988 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:15:58,020 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:15:58,289 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:15:58,301 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:15:58,301 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:15:58,312 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:15:59,013 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:15:59,462 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:15:59,471 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:15:59,879 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:15:59,979 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:16:00,833 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:16:00,835 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:16:00,838 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:16:00,841 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:16:00,843 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:16:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:05.646354+00:00 (in 4.009485 seconds) notificationworker stdout | 2023-01-17 05:16:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:01 UTC)" (scheduled at 2023-01-17 05:16:01.636362+00:00) notificationworker stdout | 2023-01-17 05:16:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:16:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:16:03,040 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:16:03,344 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:16:04,486 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:16:04,490 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:16:04,731 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:16:04,832 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:16:04,834 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:16:04,837 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:16:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:16:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:17.345601+00:00 (in 12.408328 seconds) repositorygcworker stdout | 2023-01-17 05:16:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:04 UTC)" (scheduled at 2023-01-17 05:16:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:16:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:16:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 4, 937750), True, datetime.datetime(2023, 1, 17, 5, 16, 4, 937750), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:16:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:16:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:16:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:16:05,101 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:16:05,105 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:16:05,112 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:16:05,116 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:16:05,119 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:16:05,121 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:16:05,123 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:16:05,126 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:16:05,129 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:16:05,270 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:16:05,492 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:16:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:15.646354+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 05:16:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:15 UTC)" (scheduled at 2023-01-17 05:16:05.646354+00:00) notificationworker stdout | 2023-01-17 05:16:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 5, 647274), True, datetime.datetime(2023, 1, 17, 5, 16, 5, 647274), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:15 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:16:05,709 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:16:05,749 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:16:06,236 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:16:06,795 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:16:06,887 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:16:07,060 [231] [DEBUG] [app] Starting request: urn:request:13af08b2-1de2-47a5-8ace-af9afec597e6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:07,060 [236] [DEBUG] [app] Starting request: urn:request:e250489b-424a-4fe9-9a6a-a7d72cc5f8f6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:07,065 [243] [DEBUG] [app] Starting request: urn:request:9138792b-9f63-4dce-96b8-1844fb46366f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,065 [230] [DEBUG] [app] Starting request: urn:request:a0e9ee99-4263-4ab7-ba03-dc6a28584779 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,065 [243] [DEBUG] [app] Ending request: urn:request:9138792b-9f63-4dce-96b8-1844fb46366f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9138792b-9f63-4dce-96b8-1844fb46366f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,066 [230] [DEBUG] [app] Ending request: urn:request:a0e9ee99-4263-4ab7-ba03-dc6a28584779 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0e9ee99-4263-4ab7-ba03-dc6a28584779', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:16:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:07,071 [235] [DEBUG] [app] Starting request: urn:request:b6c4ddf1-e73f-4fb0-937e-66918cf126b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:07,071 [233] [DEBUG] [app] Starting request: urn:request:85d71b22-b6c0-4b7c-a682-07372c4aed91 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:07,071 [235] [DEBUG] [app] Ending request: urn:request:b6c4ddf1-e73f-4fb0-937e-66918cf126b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6c4ddf1-e73f-4fb0-937e-66918cf126b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:07,071 [233] [DEBUG] [app] Ending request: urn:request:85d71b22-b6c0-4b7c-a682-07372c4aed91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:85d71b22-b6c0-4b7c-a682-07372c4aed91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:16:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) blobuploadcleanupworker stdout | 2023-01-17 05:16:07,085 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} gunicorn-web stdout | 2023-01-17 05:16:07,089 [231] [DEBUG] [app] Ending request: urn:request:13af08b2-1de2-47a5-8ace-af9afec597e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13af08b2-1de2-47a5-8ace-af9afec597e6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:07,089 [236] [DEBUG] [app] Ending request: urn:request:e250489b-424a-4fe9-9a6a-a7d72cc5f8f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e250489b-424a-4fe9-9a6a-a7d72cc5f8f6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:07,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:07,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:16:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 05:16:08,317 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:08,331 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:08,331 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:08,341 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:16:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:16:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:40.738568+00:00 (in 29.999526 seconds) buildlogsarchiver stdout | 2023-01-17 05:16:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:40 UTC)" (scheduled at 2023-01-17 05:16:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:16:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 16, 10, 739446), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:16:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:16:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:16:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:25.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:16:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:25 UTC)" (scheduled at 2023-01-17 05:16:15.646354+00:00) notificationworker stdout | 2023-01-17 05:16:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 15, 647196), True, datetime.datetime(2023, 1, 17, 5, 16, 15, 647196), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:16:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:19.537472+00:00 (in 2.600013 seconds) securityworker stdout | 2023-01-17 05:16:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:46 UTC)" (scheduled at 2023-01-17 05:16:16.936988+00:00) securityworker stdout | 2023-01-17 05:16:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:16:17,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:16:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:19.346120+00:00 (in 1.999982 seconds) repositorygcworker stdout | 2023-01-17 05:16:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:17 UTC)" (scheduled at 2023-01-17 05:16:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:16:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 16, 17, 346596), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:16:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 17, 346596), True, datetime.datetime(2023, 1, 17, 5, 16, 17, 346596), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:16:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 17, 346596), True, datetime.datetime(2023, 1, 17, 5, 16, 17, 346596), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 16, 17, 346596), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:16:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:16:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:17 UTC)" executed successfully builder stdout | 2023-01-17 05:16:18,351 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:18,364 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:18,364 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:18,374 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:16:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:16:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:04.936835+00:00 (in 45.590206 seconds) repositorygcworker stdout | 2023-01-17 05:16:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:19 UTC)" (scheduled at 2023-01-17 05:16:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:16:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:16:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:16:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:46.936988+00:00 (in 27.399087 seconds) securityworker stdout | 2023-01-17 05:16:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:49 UTC)" (scheduled at 2023-01-17 05:16:19.537472+00:00) securityworker stdout | 2023-01-17 05:16:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:16:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:16:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:16:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:16:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:16:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:16:19,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:16:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:16:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:51.348605+00:00 (in 29.999558 seconds) gcworker stdout | 2023-01-17 05:16:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:51 UTC)" (scheduled at 2023-01-17 05:16:21.348605+00:00) gcworker stdout | 2023-01-17 05:16:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672722981349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:16:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:16:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:16:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:16:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:16:22,059 [231] [DEBUG] [app] Starting request: urn:request:7b5bad59-626b-41a6-96fa-4b433b188d2e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:22,060 [235] [DEBUG] [app] Starting request: urn:request:3f20d783-56b9-4d59-9f7f-65eb3dfc229c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:22,065 [242] [DEBUG] [app] Starting request: urn:request:1b610cf6-41e4-4dba-b2ae-bc1da8f00d21 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,065 [230] [DEBUG] [app] Starting request: urn:request:16f07deb-74e3-49c6-aa42-1729136ec3ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,065 [242] [DEBUG] [app] Ending request: urn:request:1b610cf6-41e4-4dba-b2ae-bc1da8f00d21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b610cf6-41e4-4dba-b2ae-bc1da8f00d21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,065 [230] [DEBUG] [app] Ending request: urn:request:16f07deb-74e3-49c6-aa42-1729136ec3ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:16f07deb-74e3-49c6-aa42-1729136ec3ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:16:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:16:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:22,070 [233] [DEBUG] [app] Starting request: urn:request:9ab605a1-b57a-48d4-b884-924594b0a41e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:22,071 [231] [DEBUG] [app] Starting request: urn:request:3063a02e-36e0-4dc1-8410-02d7102c29b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:22,071 [233] [DEBUG] [app] Ending request: urn:request:9ab605a1-b57a-48d4-b884-924594b0a41e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ab605a1-b57a-48d4-b884-924594b0a41e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:22,071 [231] [DEBUG] [app] Ending request: urn:request:3063a02e-36e0-4dc1-8410-02d7102c29b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3063a02e-36e0-4dc1-8410-02d7102c29b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:16:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:16:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:22,088 [231] [DEBUG] [app] Ending request: urn:request:7b5bad59-626b-41a6-96fa-4b433b188d2e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7b5bad59-626b-41a6-96fa-4b433b188d2e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:16:22,088 [235] [DEBUG] [app] Ending request: urn:request:3f20d783-56b9-4d59-9f7f-65eb3dfc229c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f20d783-56b9-4d59-9f7f-65eb3dfc229c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:22,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:16:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:16:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:16:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:37.256652+00:00 (in 13.719173 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:16:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:23 UTC)" (scheduled at 2023-01-17 05:16:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:16:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 16, 23, 537892), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:16:23,553 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 23, 537892), True, datetime.datetime(2023, 1, 17, 5, 16, 23, 537892), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:16:23,556 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 23, 537892), True, datetime.datetime(2023, 1, 17, 5, 16, 23, 537892), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 16, 23, 537892), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:16:23,560 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:16:23,560 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:35.646354+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 05:16:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:35 UTC)" (scheduled at 2023-01-17 05:16:25.646354+00:00) notificationworker stdout | 2023-01-17 05:16:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 25, 647195), True, datetime.datetime(2023, 1, 17, 5, 16, 25, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:28,025 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:16:28,054 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:16:28,384 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:28,397 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:28,397 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:28,408 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:16:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:16:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:35.853396+00:00 (in 6.889248 seconds) exportactionlogsworker stdout | 2023-01-17 05:16:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:28 UTC)" (scheduled at 2023-01-17 05:16:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:16:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:16:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:16:29,049 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:16:29,497 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:16:29,506 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:16:29,886 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:16:30,014 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:16:30,840 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:16:30,843 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:16:30,845 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:16:30,848 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:16:30,850 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:16:33,076 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:16:33,366 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:16:34,505 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:16:34,526 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:16:34,767 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:16:34,839 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:16:34,841 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:16:34,844 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:16:35,110 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:16:35,114 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:16:35,120 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:16:35,124 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:16:35,127 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:16:35,129 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:16:35,131 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:16:35,134 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:16:35,139 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:16:35,305 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:16:35,527 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:16:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:45.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:16:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:45 UTC)" (scheduled at 2023-01-17 05:16:35.646354+00:00) notificationworker stdout | 2023-01-17 05:16:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 35, 647270), True, datetime.datetime(2023, 1, 17, 5, 16, 35, 647270), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:45 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:16:35,736 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:16:35,755 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} exportactionlogsworker stdout | 2023-01-17 05:16:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:16:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:54.963413+00:00 (in 19.109494 seconds) exportactionlogsworker stdout | 2023-01-17 05:16:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:35 UTC)" (scheduled at 2023-01-17 05:16:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:16:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:16:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 35, 854330), True, datetime.datetime(2023, 1, 17, 5, 16, 35, 854330), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:16:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:16:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:16:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:16:36,271 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:16:36,831 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:16:36,923 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:16:37,059 [231] [DEBUG] [app] Starting request: urn:request:bbdc491f-0609-45be-9ef4-7773d39c1983 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:37,059 [236] [DEBUG] [app] Starting request: urn:request:45dc2cd0-69cd-44b5-9927-7b7f64addbcf (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:37,064 [242] [DEBUG] [app] Starting request: urn:request:44970a04-2421-4040-b793-bd2ab606d452 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:37,065 [240] [DEBUG] [app] Starting request: urn:request:8a8c95df-89b9-448d-bd97-dba6d0c69be6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:37,065 [242] [DEBUG] [app] Ending request: urn:request:44970a04-2421-4040-b793-bd2ab606d452 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44970a04-2421-4040-b793-bd2ab606d452', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:37,065 [240] [DEBUG] [app] Ending request: urn:request:8a8c95df-89b9-448d-bd97-dba6d0c69be6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a8c95df-89b9-448d-bd97-dba6d0c69be6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:37,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:16:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:16:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:37,070 [235] [DEBUG] [app] Starting request: urn:request:27e2b0c7-b1c1-435f-aac4-194d92b12e34 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:37,070 [233] [DEBUG] [app] Starting request: urn:request:c39ea954-844a-4fa1-9b42-a5d51b18889e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:37,070 [235] [DEBUG] [app] Ending request: urn:request:27e2b0c7-b1c1-435f-aac4-194d92b12e34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27e2b0c7-b1c1-435f-aac4-194d92b12e34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:16:37,071 [233] [DEBUG] [app] Ending request: urn:request:c39ea954-844a-4fa1-9b42-a5d51b18889e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c39ea954-844a-4fa1-9b42-a5d51b18889e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:16:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:37,088 [231] [DEBUG] [app] Ending request: urn:request:bbdc491f-0609-45be-9ef4-7773d39c1983 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bbdc491f-0609-45be-9ef4-7773d39c1983', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:37,088 [236] [DEBUG] [app] Ending request: urn:request:45dc2cd0-69cd-44b5-9927-7b7f64addbcf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:45dc2cd0-69cd-44b5-9927-7b7f64addbcf', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:16:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) blobuploadcleanupworker stdout | 2023-01-17 05:16:37,118 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:16:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:16:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:37.537407+00:00 (in 0.280174 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:16:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:37 UTC)" (scheduled at 2023-01-17 05:16:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:16:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:16:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 37, 257733), True, datetime.datetime(2023, 1, 17, 5, 16, 37, 257733), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:16:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:16:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:16:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:16:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:16:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:37.256652+00:00 (in 59.718773 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:16:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:37 UTC)" (scheduled at 2023-01-17 05:16:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:16:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:16:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:37 UTC)" executed successfully builder stdout | 2023-01-17 05:16:38,418 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:38,431 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:38,431 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:38,441 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:16:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:16:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:10.738568+00:00 (in 29.999504 seconds) buildlogsarchiver stdout | 2023-01-17 05:16:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:10 UTC)" (scheduled at 2023-01-17 05:16:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:16:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 16, 40, 739473), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:16:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:16:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:16:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:16:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:16:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:42.941536+00:00 (in 1.792760 seconds) namespacegcworker stdout | 2023-01-17 05:16:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:41 UTC)" (scheduled at 2023-01-17 05:16:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:16:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:16:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 41, 149242), True, datetime.datetime(2023, 1, 17, 5, 16, 41, 149242), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:16:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:16:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:16:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:16:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:16:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:06.941185+00:00 (in 23.999166 seconds) namespacegcworker stdout | 2023-01-17 05:16:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:42 UTC)" (scheduled at 2023-01-17 05:16:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:16:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:16:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:17:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:55.646354+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 05:16:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:55 UTC)" (scheduled at 2023-01-17 05:16:45.646354+00:00) notificationworker stdout | 2023-01-17 05:16:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 45, 647248), True, datetime.datetime(2023, 1, 17, 5, 16, 45, 647248), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:16:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:16:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:16:49.537472+00:00 (in 2.600031 seconds) securityworker stdout | 2023-01-17 05:16:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:16 UTC)" (scheduled at 2023-01-17 05:16:46.936988+00:00) securityworker stdout | 2023-01-17 05:16:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:16 UTC)" executed successfully builder stdout | 2023-01-17 05:16:48,452 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:48,464 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:48,464 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:48,476 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:16:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:16.936988+00:00 (in 27.399045 seconds) securityworker stdout | 2023-01-17 05:16:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:19 UTC)" (scheduled at 2023-01-17 05:16:49.537472+00:00) securityworker stdout | 2023-01-17 05:16:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:16:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:16:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:16:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:16:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:16:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:16:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:16:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:16:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:16:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:16:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:16:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:16:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:16:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:16:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:16:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:21.348605+00:00 (in 29.999425 seconds) gcworker stdout | 2023-01-17 05:16:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:21 UTC)" (scheduled at 2023-01-17 05:16:51.348605+00:00) gcworker stdout | 2023-01-17 05:16:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723011349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:16:51,372 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:16:51,373 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:16:51,373 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:16:52,059 [231] [DEBUG] [app] Starting request: urn:request:d78b4bea-db78-4fed-ba8a-95a4642bc24b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:52,059 [236] [DEBUG] [app] Starting request: urn:request:a64b67f1-499f-45ca-bbb1-00c2efcb94d5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:16:52,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,062 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:16:52,063 [230] [DEBUG] [app] Starting request: urn:request:824fdd53-8cb0-49a7-a8f7-6f6fbb6968a1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:52,063 [243] [DEBUG] [app] Starting request: urn:request:404152a4-bd14-45f2-99f6-bfe443adee14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:16:52,064 [230] [DEBUG] [app] Ending request: urn:request:824fdd53-8cb0-49a7-a8f7-6f6fbb6968a1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:824fdd53-8cb0-49a7-a8f7-6f6fbb6968a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:52,064 [243] [DEBUG] [app] Ending request: urn:request:404152a4-bd14-45f2-99f6-bfe443adee14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:404152a4-bd14-45f2-99f6-bfe443adee14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:16:52,064 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:52,064 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:16:52,064 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:52,064 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:52,065 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:16:52,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:16:52,069 [235] [DEBUG] [app] Starting request: urn:request:17a29894-614c-4bca-b1e4-068b05781398 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:52,069 [233] [DEBUG] [app] Starting request: urn:request:89c35f13-20cd-4555-a1a2-0409415f0879 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:16:52,070 [235] [DEBUG] [app] Ending request: urn:request:17a29894-614c-4bca-b1e4-068b05781398 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17a29894-614c-4bca-b1e4-068b05781398', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:16:52,070 [233] [DEBUG] [app] Ending request: urn:request:89c35f13-20cd-4555-a1a2-0409415f0879 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89c35f13-20cd-4555-a1a2-0409415f0879', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:16:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:16:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:52,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:16:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:16:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:16:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:52,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:16:52,086 [231] [DEBUG] [app] Ending request: urn:request:d78b4bea-db78-4fed-ba8a-95a4642bc24b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d78b4bea-db78-4fed-ba8a-95a4642bc24b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:52,086 [236] [DEBUG] [app] Ending request: urn:request:a64b67f1-499f-45ca-bbb1-00c2efcb94d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a64b67f1-499f-45ca-bbb1-00c2efcb94d5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:16:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:16:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:16:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 05:16:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:16:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:28.963662+00:00 (in 33.999770 seconds) exportactionlogsworker stdout | 2023-01-17 05:16:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:54 UTC)" (scheduled at 2023-01-17 05:16:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:16:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 16, 54, 964353), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:16:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 54, 964353), True, datetime.datetime(2023, 1, 17, 5, 16, 54, 964353), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:16:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 16, 54, 964353), True, datetime.datetime(2023, 1, 17, 5, 16, 54, 964353), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 16, 54, 964353), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:16:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:16:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:21:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:16:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:01.636362+00:00 (in 5.989524 seconds) notificationworker stdout | 2023-01-17 05:16:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:05 UTC)" (scheduled at 2023-01-17 05:16:55.646354+00:00) notificationworker stdout | 2023-01-17 05:16:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:16:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 16, 55, 647255), True, datetime.datetime(2023, 1, 17, 5, 16, 55, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:16:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:16:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:16:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:16:58,060 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:16:58,088 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:16:58,486 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:16:58,499 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:16:58,499 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:16:58,510 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:16:59,085 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:16:59,533 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:16:59,541 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:16:59,892 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:17:00,049 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:17:00,848 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:17:00,851 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:17:00,853 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:17:00,855 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:17:00,858 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:17:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:05.646354+00:00 (in 4.009493 seconds) notificationworker stdout | 2023-01-17 05:17:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:01 UTC)" (scheduled at 2023-01-17 05:17:01.636362+00:00) notificationworker stdout | 2023-01-17 05:17:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:17:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:17:03,092 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:17:03,405 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:17:04,537 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:17:04,540 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:17:04,794 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:17:04,845 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:17:04,848 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:17:04,850 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:17:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:17:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:19.346120+00:00 (in 14.408907 seconds) repositorygcworker stdout | 2023-01-17 05:17:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:04 UTC)" (scheduled at 2023-01-17 05:17:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:17:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:17:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 4, 937631), True, datetime.datetime(2023, 1, 17, 5, 17, 4, 937631), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:17:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:17:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:17:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:17:05,119 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:17:05,122 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:17:05,125 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:17:05,131 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:17:05,133 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:17:05,136 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:17:05,139 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:17:05,141 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:17:05,149 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:17:05,340 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:17:05,539 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:17:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:15.646354+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:17:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:15 UTC)" (scheduled at 2023-01-17 05:17:05.646354+00:00) notificationworker stdout | 2023-01-17 05:17:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 5, 647351), True, datetime.datetime(2023, 1, 17, 5, 17, 5, 647351), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:15 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:17:05,757 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:17:05,782 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:17:06,307 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:17:06,866 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} namespacegcworker stdout | 2023-01-17 05:17:06,942 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:17:06,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:41.148245+00:00 (in 34.205724 seconds) namespacegcworker stdout | 2023-01-17 05:17:06,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:22:06 UTC)" (scheduled at 2023-01-17 05:17:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:17:06,943 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 17, 6, 942922), 'namespacegc/%']) exportactionlogsworker stdout | 2023-01-17 05:17:06,946 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} namespacegcworker stdout | 2023-01-17 05:17:06,954 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 17, 6, 942922), True, datetime.datetime(2023, 1, 17, 5, 17, 6, 942922), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:17:06,958 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 17, 6, 942922), True, datetime.datetime(2023, 1, 17, 5, 17, 6, 942922), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 17, 6, 942922), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:17:06,961 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:17:06,961 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:22:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:17:07,060 [231] [DEBUG] [app] Starting request: urn:request:5b05f465-d1b6-4e94-95d1-57f19d920d91 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:07,060 [233] [DEBUG] [app] Starting request: urn:request:af6ea9c8-87e2-42c7-9b67-1ee674a5bf0d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:07,064 [242] [DEBUG] [app] Starting request: urn:request:5af59fce-d928-4d42-99aa-d32f9f7a0b30 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:07,065 [243] [DEBUG] [app] Starting request: urn:request:31a7c408-1ef3-41aa-b0a0-7c32597133e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:07,065 [242] [DEBUG] [app] Ending request: urn:request:5af59fce-d928-4d42-99aa-d32f9f7a0b30 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5af59fce-d928-4d42-99aa-d32f9f7a0b30', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:17:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:17:07,065 [243] [DEBUG] [app] Ending request: urn:request:31a7c408-1ef3-41aa-b0a0-7c32597133e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31a7c408-1ef3-41aa-b0a0-7c32597133e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:17:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:17:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:07,070 [236] [DEBUG] [app] Starting request: urn:request:e905c4ae-2658-40f4-8514-1ef4cba3e80e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:07,070 [233] [DEBUG] [app] Starting request: urn:request:75705f80-c88a-42be-bfe6-1580ad2198a8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:07,071 [236] [DEBUG] [app] Ending request: urn:request:e905c4ae-2658-40f4-8514-1ef4cba3e80e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e905c4ae-2658-40f4-8514-1ef4cba3e80e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:07,071 [233] [DEBUG] [app] Ending request: urn:request:75705f80-c88a-42be-bfe6-1580ad2198a8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:75705f80-c88a-42be-bfe6-1580ad2198a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:07,087 [231] [DEBUG] [app] Ending request: urn:request:5b05f465-d1b6-4e94-95d1-57f19d920d91 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5b05f465-d1b6-4e94-95d1-57f19d920d91', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:07,087 [233] [DEBUG] [app] Ending request: urn:request:af6ea9c8-87e2-42c7-9b67-1ee674a5bf0d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af6ea9c8-87e2-42c7-9b67-1ee674a5bf0d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:17:07,087 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:17:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) blobuploadcleanupworker stdout | 2023-01-17 05:17:07,154 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:17:08,511 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:08,523 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:08,523 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:08,534 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:17:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:17:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:40.738568+00:00 (in 29.999515 seconds) buildlogsarchiver stdout | 2023-01-17 05:17:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:40 UTC)" (scheduled at 2023-01-17 05:17:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:17:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 17, 10, 739478), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:17:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:17:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:17:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:25.646354+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 05:17:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:25 UTC)" (scheduled at 2023-01-17 05:17:15.646354+00:00) notificationworker stdout | 2023-01-17 05:17:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 15, 647193), True, datetime.datetime(2023, 1, 17, 5, 17, 15, 647193), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:17:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:19.537472+00:00 (in 2.599984 seconds) securityworker stdout | 2023-01-17 05:17:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:46 UTC)" (scheduled at 2023-01-17 05:17:16.936988+00:00) securityworker stdout | 2023-01-17 05:17:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:46 UTC)" executed successfully builder stdout | 2023-01-17 05:17:18,544 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:18,557 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:18,557 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:18,568 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:17:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:17:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:04.936835+00:00 (in 45.590273 seconds) repositorygcworker stdout | 2023-01-17 05:17:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:19 UTC)" (scheduled at 2023-01-17 05:17:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:17:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:17:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:17:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:46.936988+00:00 (in 27.398988 seconds) securityworker stdout | 2023-01-17 05:17:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:49 UTC)" (scheduled at 2023-01-17 05:17:19.537472+00:00) securityworker stdout | 2023-01-17 05:17:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:17:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:17:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:17:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:17:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:17:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:17:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:17:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:17:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:51.348605+00:00 (in 29.999302 seconds) gcworker stdout | 2023-01-17 05:17:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:51 UTC)" (scheduled at 2023-01-17 05:17:21.348605+00:00) gcworker stdout | 2023-01-17 05:17:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723041349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:17:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:17:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:17:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:17:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:17:22,060 [231] [DEBUG] [app] Starting request: urn:request:74a422b0-c02d-4b73-acbe-0d6aa4e57487 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:22,060 [236] [DEBUG] [app] Starting request: urn:request:d108249b-c727-4fcd-9085-adf627e7e760 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:22,065 [242] [DEBUG] [app] Starting request: urn:request:3f4defe8-2fed-4283-ad1b-5970ee0fdc01 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:22,066 [230] [DEBUG] [app] Starting request: urn:request:f329e4b6-d683-49c6-bfb0-64f2aa4754db (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:22,066 [242] [DEBUG] [app] Ending request: urn:request:3f4defe8-2fed-4283-ad1b-5970ee0fdc01 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f4defe8-2fed-4283-ad1b-5970ee0fdc01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:22,066 [230] [DEBUG] [app] Ending request: urn:request:f329e4b6-d683-49c6-bfb0-64f2aa4754db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f329e4b6-d683-49c6-bfb0-64f2aa4754db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:17:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:17:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:22,071 [235] [DEBUG] [app] Starting request: urn:request:a41e4e77-e80d-4f5b-af2b-ee41dbf3d3e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:22,071 [233] [DEBUG] [app] Starting request: urn:request:8df2e61f-3729-4f97-9c3d-0642c0119c1e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:22,071 [235] [DEBUG] [app] Ending request: urn:request:a41e4e77-e80d-4f5b-af2b-ee41dbf3d3e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a41e4e77-e80d-4f5b-af2b-ee41dbf3d3e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:17:22,072 [233] [DEBUG] [app] Ending request: urn:request:8df2e61f-3729-4f97-9c3d-0642c0119c1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8df2e61f-3729-4f97-9c3d-0642c0119c1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:22,085 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:22,085 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:22,095 [236] [DEBUG] [app] Ending request: urn:request:d108249b-c727-4fcd-9085-adf627e7e760 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d108249b-c727-4fcd-9085-adf627e7e760', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:22,095 [231] [DEBUG] [app] Ending request: urn:request:74a422b0-c02d-4b73-acbe-0d6aa4e57487 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:74a422b0-c02d-4b73-acbe-0d6aa4e57487', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:22,095 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:22,095 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:22,095 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:17:22,095 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) notificationworker stdout | 2023-01-17 05:17:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:35.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:17:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:35 UTC)" (scheduled at 2023-01-17 05:17:25.646354+00:00) notificationworker stdout | 2023-01-17 05:17:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 25, 647243), True, datetime.datetime(2023, 1, 17, 5, 17, 25, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:28,090 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:17:28,123 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:17:28,578 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:28,591 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:28,591 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:28,602 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:17:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:17:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:35.853396+00:00 (in 6.889216 seconds) exportactionlogsworker stdout | 2023-01-17 05:17:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:28 UTC)" (scheduled at 2023-01-17 05:17:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:17:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:17:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:17:29,121 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:17:29,539 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:17:29,560 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:17:29,902 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:17:30,054 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:17:30,856 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:17:30,859 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:17:30,862 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:17:30,864 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:17:30,866 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:17:33,122 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:17:33,440 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:17:34,550 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:17:34,568 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:17:34,805 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:17:34,851 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:17:34,855 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:17:34,857 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:17:35,128 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:17:35,130 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:17:35,132 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:17:35,139 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:17:35,141 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:17:35,144 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:17:35,146 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:17:35,148 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:17:35,156 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:17:35,350 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:17:35,574 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:17:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:45.646354+00:00 (in 9.999565 seconds) notificationworker stdout | 2023-01-17 05:17:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:45 UTC)" (scheduled at 2023-01-17 05:17:35.646354+00:00) notificationworker stdout | 2023-01-17 05:17:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 35, 647203), True, datetime.datetime(2023, 1, 17, 5, 17, 35, 647203), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:45 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:17:35,792 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:17:35,796 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} exportactionlogsworker stdout | 2023-01-17 05:17:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:17:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:28.963662+00:00 (in 53.109673 seconds) exportactionlogsworker stdout | 2023-01-17 05:17:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:35 UTC)" (scheduled at 2023-01-17 05:17:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:17:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:17:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 35, 854392), True, datetime.datetime(2023, 1, 17, 5, 17, 35, 854392), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:17:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:17:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:17:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:17:36,328 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:17:36,902 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:17:36,952 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:17:37,059 [235] [DEBUG] [app] Starting request: urn:request:8485a8be-9949-491b-bd60-9aebebdf9991 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:37,060 [231] [DEBUG] [app] Starting request: urn:request:0f0644d5-55e5-41c4-8ab9-cabc1dcbc12c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:37,064 [241] [DEBUG] [app] Starting request: urn:request:4da75461-8efe-472a-a869-f71dee69db40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:37,064 [241] [DEBUG] [app] Ending request: urn:request:4da75461-8efe-472a-a869-f71dee69db40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4da75461-8efe-472a-a869-f71dee69db40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:37,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:17:37,065 [243] [DEBUG] [app] Starting request: urn:request:31781843-23fb-4623-b694-43c89ce43a2b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:17:37,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:17:37,065 [243] [DEBUG] [app] Ending request: urn:request:31781843-23fb-4623-b694-43c89ce43a2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31781843-23fb-4623-b694-43c89ce43a2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:17:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:37,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:37,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:37,069 [233] [DEBUG] [app] Starting request: urn:request:1afa46be-b142-4a77-8cf5-6b7f82264b53 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:37,069 [233] [DEBUG] [app] Ending request: urn:request:1afa46be-b142-4a77-8cf5-6b7f82264b53 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1afa46be-b142-4a77-8cf5-6b7f82264b53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:17:37,069 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:37,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:37,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:37,071 [236] [DEBUG] [app] Starting request: urn:request:79ef1ff7-331e-412e-863b-73a260ac90ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:37,071 [236] [DEBUG] [app] Ending request: urn:request:79ef1ff7-331e-412e-863b-73a260ac90ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79ef1ff7-331e-412e-863b-73a260ac90ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:37,086 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:37,089 [235] [DEBUG] [app] Ending request: urn:request:8485a8be-9949-491b-bd60-9aebebdf9991 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8485a8be-9949-491b-bd60-9aebebdf9991', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:37,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 05:17:37,095 [231] [DEBUG] [app] Ending request: urn:request:0f0644d5-55e5-41c4-8ab9-cabc1dcbc12c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f0644d5-55e5-41c4-8ab9-cabc1dcbc12c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:37,095 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:37,096 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.037) blobuploadcleanupworker stdout | 2023-01-17 05:17:37,171 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:17:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:17:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:37.537407+00:00 (in 0.280297 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:17:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:37 UTC)" (scheduled at 2023-01-17 05:17:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:17:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:17:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 37, 257537), True, datetime.datetime(2023, 1, 17, 5, 17, 37, 257537), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:17:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:17:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:17:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:17:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:17:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:37.256652+00:00 (in 59.718835 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:17:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:37 UTC)" (scheduled at 2023-01-17 05:17:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:17:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:17:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:37 UTC)" executed successfully builder stdout | 2023-01-17 05:17:38,612 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:38,625 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:38,625 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:38,635 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:17:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:17:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:10.738568+00:00 (in 29.999511 seconds) buildlogsarchiver stdout | 2023-01-17 05:17:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:10 UTC)" (scheduled at 2023-01-17 05:17:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:17:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 17, 40, 739434), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:17:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:17:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:17:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:17:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:17:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:42.941536+00:00 (in 1.792768 seconds) namespacegcworker stdout | 2023-01-17 05:17:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:41 UTC)" (scheduled at 2023-01-17 05:17:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:17:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:17:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 41, 149201), True, datetime.datetime(2023, 1, 17, 5, 17, 41, 149201), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:17:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:17:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:17:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:17:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:17:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:41.148245+00:00 (in 58.206229 seconds) namespacegcworker stdout | 2023-01-17 05:17:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:42 UTC)" (scheduled at 2023-01-17 05:17:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:17:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:17:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:18:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:55.646354+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 05:17:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:55 UTC)" (scheduled at 2023-01-17 05:17:45.646354+00:00) notificationworker stdout | 2023-01-17 05:17:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 45, 647292), True, datetime.datetime(2023, 1, 17, 5, 17, 45, 647292), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:17:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:17:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:17:49.537472+00:00 (in 2.599977 seconds) securityworker stdout | 2023-01-17 05:17:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:16 UTC)" (scheduled at 2023-01-17 05:17:46.936988+00:00) securityworker stdout | 2023-01-17 05:17:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:46,953 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:46,956 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:46,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:16 UTC)" executed successfully builder stdout | 2023-01-17 05:17:48,645 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:48,658 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:48,658 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:48,669 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:17:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:16.936988+00:00 (in 27.399010 seconds) securityworker stdout | 2023-01-17 05:17:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:19 UTC)" (scheduled at 2023-01-17 05:17:49.537472+00:00) securityworker stdout | 2023-01-17 05:17:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:17:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:17:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:17:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:17:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:17:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:17:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:17:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:17:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:17:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:17:49,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:17:49,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:17:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:17:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:17:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:17:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:21.348605+00:00 (in 29.999488 seconds) gcworker stdout | 2023-01-17 05:17:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:21 UTC)" (scheduled at 2023-01-17 05:17:51.348605+00:00) gcworker stdout | 2023-01-17 05:17:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723071349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:17:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:17:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:17:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:17:52,060 [231] [DEBUG] [app] Starting request: urn:request:4921d9a0-0138-4a94-9936-3f59382ea890 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:52,060 [233] [DEBUG] [app] Starting request: urn:request:0f973d5a-a0bf-4fdc-99b6-31f5054705ba (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:17:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:17:52,064 [241] [DEBUG] [app] Starting request: urn:request:a3479c5b-efb6-41ca-afed-c1ce3d14b6e5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:52,065 [230] [DEBUG] [app] Starting request: urn:request:107a204d-d71a-477e-970b-53b12f7c332c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:17:52,065 [241] [DEBUG] [app] Ending request: urn:request:a3479c5b-efb6-41ca-afed-c1ce3d14b6e5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3479c5b-efb6-41ca-afed-c1ce3d14b6e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:17:52,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:17:52,065 [230] [DEBUG] [app] Ending request: urn:request:107a204d-d71a-477e-970b-53b12f7c332c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:107a204d-d71a-477e-970b-53b12f7c332c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:17:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:17:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:17:52,070 [236] [DEBUG] [app] Starting request: urn:request:7a309547-de76-4cc4-8d10-37f307d2bc34 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:52,071 [235] [DEBUG] [app] Starting request: urn:request:7c028eac-bed3-4082-92a1-934d40d33d0a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:17:52,071 [236] [DEBUG] [app] Ending request: urn:request:7a309547-de76-4cc4-8d10-37f307d2bc34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a309547-de76-4cc4-8d10-37f307d2bc34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:17:52,071 [235] [DEBUG] [app] Ending request: urn:request:7c028eac-bed3-4082-92a1-934d40d33d0a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c028eac-bed3-4082-92a1-934d40d33d0a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:17:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:17:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:17:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:17:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:17:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:17:52,087 [231] [DEBUG] [app] Ending request: urn:request:4921d9a0-0138-4a94-9936-3f59382ea890 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4921d9a0-0138-4a94-9936-3f59382ea890', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:52,088 [233] [DEBUG] [app] Ending request: urn:request:0f973d5a-a0bf-4fdc-99b6-31f5054705ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f973d5a-a0bf-4fdc-99b6-31f5054705ba', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:17:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:17:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:17:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:17:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:17:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:01.636362+00:00 (in 5.989476 seconds) notificationworker stdout | 2023-01-17 05:17:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:05 UTC)" (scheduled at 2023-01-17 05:17:55.646354+00:00) notificationworker stdout | 2023-01-17 05:17:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:17:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 17, 55, 647290), True, datetime.datetime(2023, 1, 17, 5, 17, 55, 647290), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:17:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:17:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:17:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:17:58,118 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:17:58,158 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:17:58,679 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:17:58,692 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:17:58,692 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:17:58,702 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:17:59,150 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:17:59,571 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:17:59,573 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:17:59,908 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:18:00,060 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:18:00,864 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:18:00,866 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:18:00,869 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:18:00,871 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:18:00,874 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:18:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:05.646354+00:00 (in 4.009541 seconds) notificationworker stdout | 2023-01-17 05:18:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:01 UTC)" (scheduled at 2023-01-17 05:18:01.636362+00:00) notificationworker stdout | 2023-01-17 05:18:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:18:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:18:03,158 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:18:03,475 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:18:04,558 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:18:04,583 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:18:04,813 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:18:04,857 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:18:04,861 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:18:04,864 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:18:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:18:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:19.346120+00:00 (in 14.408897 seconds) repositorygcworker stdout | 2023-01-17 05:18:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:04 UTC)" (scheduled at 2023-01-17 05:18:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:18:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:18:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 4, 937653), True, datetime.datetime(2023, 1, 17, 5, 18, 4, 937653), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:18:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:18:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:18:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:18:05,136 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:18:05,139 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:18:05,141 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:18:05,145 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:18:05,148 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:18:05,151 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:18:05,157 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:18:05,159 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:18:05,165 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:18:05,368 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:18:05,609 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:18:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:15.646354+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 05:18:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:15 UTC)" (scheduled at 2023-01-17 05:18:05.646354+00:00) notificationworker stdout | 2023-01-17 05:18:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 5, 647162), True, datetime.datetime(2023, 1, 17, 5, 18, 5, 647162), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:15 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:18:05,826 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:18:05,828 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:18:06,339 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:18:06,907 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:18:06,958 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:18:07,060 [236] [DEBUG] [app] Starting request: urn:request:1402ea4c-be1e-406c-b58d-b705f324c2ef (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:07,060 [235] [DEBUG] [app] Starting request: urn:request:3e76ae5d-1cbf-4933-93dc-09ceec834a2e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:07,065 [242] [DEBUG] [app] Starting request: urn:request:4d691871-a715-4a36-a6dd-55b6e24e572a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:07,065 [230] [DEBUG] [app] Starting request: urn:request:c737c234-4949-44f1-bdde-a97421ebc2a6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:07,065 [242] [DEBUG] [app] Ending request: urn:request:4d691871-a715-4a36-a6dd-55b6e24e572a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d691871-a715-4a36-a6dd-55b6e24e572a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:18:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:18:07,065 [230] [DEBUG] [app] Ending request: urn:request:c737c234-4949-44f1-bdde-a97421ebc2a6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c737c234-4949-44f1-bdde-a97421ebc2a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:18:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:18:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:07,071 [231] [DEBUG] [app] Starting request: urn:request:2fe2d79f-ff08-4566-b05d-de73ced6179f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:07,071 [236] [DEBUG] [app] Starting request: urn:request:a364d6d1-4104-4ae7-9a40-c66373342042 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:07,071 [231] [DEBUG] [app] Ending request: urn:request:2fe2d79f-ff08-4566-b05d-de73ced6179f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2fe2d79f-ff08-4566-b05d-de73ced6179f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:07,071 [236] [DEBUG] [app] Ending request: urn:request:a364d6d1-4104-4ae7-9a40-c66373342042 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a364d6d1-4104-4ae7-9a40-c66373342042', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:18:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:07,088 [236] [DEBUG] [app] Ending request: urn:request:1402ea4c-be1e-406c-b58d-b705f324c2ef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1402ea4c-be1e-406c-b58d-b705f324c2ef', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:07,088 [235] [DEBUG] [app] Ending request: urn:request:3e76ae5d-1cbf-4933-93dc-09ceec834a2e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3e76ae5d-1cbf-4933-93dc-09ceec834a2e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:18:07,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" blobuploadcleanupworker stdout | 2023-01-17 05:18:07,194 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:18:08,713 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:08,725 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:08,725 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:08,736 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:18:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:18:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:40.738568+00:00 (in 29.999518 seconds) buildlogsarchiver stdout | 2023-01-17 05:18:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:40 UTC)" (scheduled at 2023-01-17 05:18:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:18:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 18, 10, 739510), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:18:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:18:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:18:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:25.646354+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 05:18:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:25 UTC)" (scheduled at 2023-01-17 05:18:15.646354+00:00) notificationworker stdout | 2023-01-17 05:18:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 15, 647243), True, datetime.datetime(2023, 1, 17, 5, 18, 15, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:18:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:19.537472+00:00 (in 2.600016 seconds) securityworker stdout | 2023-01-17 05:18:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:46 UTC)" (scheduled at 2023-01-17 05:18:16.936988+00:00) securityworker stdout | 2023-01-17 05:18:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:46 UTC)" executed successfully builder stdout | 2023-01-17 05:18:18,746 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:18,758 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:18,758 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:18,768 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:18:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:18:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:04.936835+00:00 (in 45.590251 seconds) repositorygcworker stdout | 2023-01-17 05:18:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:19 UTC)" (scheduled at 2023-01-17 05:18:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:18:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:18:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:18:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:46.936988+00:00 (in 27.399016 seconds) securityworker stdout | 2023-01-17 05:18:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:49 UTC)" (scheduled at 2023-01-17 05:18:19.537472+00:00) securityworker stdout | 2023-01-17 05:18:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:18:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:18:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:18:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:18:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:18:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:18:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:18:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:18:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:51.348605+00:00 (in 29.999510 seconds) gcworker stdout | 2023-01-17 05:18:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:51 UTC)" (scheduled at 2023-01-17 05:18:21.348605+00:00) gcworker stdout | 2023-01-17 05:18:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723101349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:18:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:18:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:18:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:18:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:18:22,060 [233] [DEBUG] [app] Starting request: urn:request:3aed9921-1362-42bc-b44a-1646c8d08ef0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:22,061 [235] [DEBUG] [app] Starting request: urn:request:ccd3ef8d-a6ee-4feb-9059-e3ac67672b10 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:22,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:22,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:22,066 [230] [DEBUG] [app] Starting request: urn:request:56350b8a-edac-4952-8e3b-2b63bc044515 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:22,066 [242] [DEBUG] [app] Starting request: urn:request:59aa8ecf-a3e0-4af4-aeca-9e2d008e9956 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:22,066 [230] [DEBUG] [app] Ending request: urn:request:56350b8a-edac-4952-8e3b-2b63bc044515 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56350b8a-edac-4952-8e3b-2b63bc044515', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:22,067 [242] [DEBUG] [app] Ending request: urn:request:59aa8ecf-a3e0-4af4-aeca-9e2d008e9956 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59aa8ecf-a3e0-4af4-aeca-9e2d008e9956', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:22,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:18:22,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:22,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:22,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:22,071 [231] [DEBUG] [app] Starting request: urn:request:4451d9e6-da90-4b39-a277-416f07706e9f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:22,071 [236] [DEBUG] [app] Starting request: urn:request:a2952336-b46f-471e-b71a-3188b256906f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:22,071 [231] [DEBUG] [app] Ending request: urn:request:4451d9e6-da90-4b39-a277-416f07706e9f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4451d9e6-da90-4b39-a277-416f07706e9f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:22,071 [236] [DEBUG] [app] Ending request: urn:request:a2952336-b46f-471e-b71a-3188b256906f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a2952336-b46f-471e-b71a-3188b256906f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:22,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:22,088 [233] [DEBUG] [app] Ending request: urn:request:3aed9921-1362-42bc-b44a-1646c8d08ef0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3aed9921-1362-42bc-b44a-1646c8d08ef0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:22,088 [235] [DEBUG] [app] Ending request: urn:request:ccd3ef8d-a6ee-4feb-9059-e3ac67672b10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ccd3ef8d-a6ee-4feb-9059-e3ac67672b10', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:22,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:18:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:18:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:35.646354+00:00 (in 9.999486 seconds) notificationworker stdout | 2023-01-17 05:18:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:35 UTC)" (scheduled at 2023-01-17 05:18:25.646354+00:00) notificationworker stdout | 2023-01-17 05:18:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 25, 647344), True, datetime.datetime(2023, 1, 17, 5, 18, 25, 647344), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:28,131 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:18:28,194 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:18:28,777 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:28,789 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:28,789 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:28,799 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:18:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:18:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:35.853396+00:00 (in 6.889262 seconds) exportactionlogsworker stdout | 2023-01-17 05:18:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:28 UTC)" (scheduled at 2023-01-17 05:18:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:18:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:18:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:18:29,185 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:18:29,580 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:18:29,608 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:18:29,944 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:18:30,086 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:18:30,872 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:18:30,874 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:18:30,877 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:18:30,879 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:18:30,881 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:18:33,173 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:18:33,487 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:18:34,573 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:18:34,600 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:18:34,848 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-secscan stdout | 2023-01-17 05:18:34,862 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:18:34,867 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:18:34,870 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:18:35,142 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:18:35,145 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:18:35,151 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:18:35,153 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:18:35,158 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:18:35,160 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:18:35,167 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:18:35,170 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:18:35,173 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:18:35,404 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:18:35,628 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:18:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:36.635997+00:00 (in 0.989132 seconds) notificationworker stdout | 2023-01-17 05:18:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:45 UTC)" (scheduled at 2023-01-17 05:18:35.646354+00:00) notificationworker stdout | 2023-01-17 05:18:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 35, 647258), True, datetime.datetime(2023, 1, 17, 5, 18, 35, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:18:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:18:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:28.963662+00:00 (in 53.109793 seconds) exportactionlogsworker stdout | 2023-01-17 05:18:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:35 UTC)" (scheduled at 2023-01-17 05:18:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:18:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:18:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 35, 854229), True, datetime.datetime(2023, 1, 17, 5, 18, 35, 854229), 0, 'exportactionlogs/%', 50, 1, 0]) repositoryactioncounter stdout | 2023-01-17 05:18:35,856 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:18:35,863 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} exportactionlogsworker stdout | 2023-01-17 05:18:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:18:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:18:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:35 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:18:36,376 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} notificationworker stdout | 2023-01-17 05:18:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:45.646354+00:00 (in 9.009906 seconds) notificationworker stdout | 2023-01-17 05:18:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:23:36 UTC)" (scheduled at 2023-01-17 05:18:36.635997+00:00) notificationworker stdout | 2023-01-17 05:18:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 18, 36, 636854), 'notification/%']) notificationworker stdout | 2023-01-17 05:18:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 18, 36, 636854), True, datetime.datetime(2023, 1, 17, 5, 18, 36, 636854), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:18:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 18, 36, 636854), True, datetime.datetime(2023, 1, 17, 5, 18, 36, 636854), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 18, 36, 636854), 'notification/%']) notificationworker stdout | 2023-01-17 05:18:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:23:36 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:18:36,937 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:18:36,993 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:18:37,060 [231] [DEBUG] [app] Starting request: urn:request:7f3a89e8-42bc-4fa4-bdb7-733da5a8e45d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:37,060 [236] [DEBUG] [app] Starting request: urn:request:1ed879a1-091d-43c0-bb09-cca75535c3de (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:37,065 [230] [DEBUG] [app] Starting request: urn:request:ac267bf2-fd9e-4e0f-a197-2736ef595511 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,066 [242] [DEBUG] [app] Starting request: urn:request:471785c3-ff0c-47cd-ab0b-96c460147eed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,066 [230] [DEBUG] [app] Ending request: urn:request:ac267bf2-fd9e-4e0f-a197-2736ef595511 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac267bf2-fd9e-4e0f-a197-2736ef595511', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,066 [242] [DEBUG] [app] Ending request: urn:request:471785c3-ff0c-47cd-ab0b-96c460147eed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:471785c3-ff0c-47cd-ab0b-96c460147eed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:18:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:18:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:37,070 [235] [DEBUG] [app] Starting request: urn:request:04a82fb6-c2f7-4e91-8475-9d4e80f3cb1d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:37,070 [231] [DEBUG] [app] Starting request: urn:request:cb8f7d51-d8e2-45e5-bebd-17036a61544e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:37,071 [235] [DEBUG] [app] Ending request: urn:request:04a82fb6-c2f7-4e91-8475-9d4e80f3cb1d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:04a82fb6-c2f7-4e91-8475-9d4e80f3cb1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:37,071 [231] [DEBUG] [app] Ending request: urn:request:cb8f7d51-d8e2-45e5-bebd-17036a61544e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cb8f7d51-d8e2-45e5-bebd-17036a61544e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:18:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:37,088 [231] [DEBUG] [app] Ending request: urn:request:7f3a89e8-42bc-4fa4-bdb7-733da5a8e45d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f3a89e8-42bc-4fa4-bdb7-733da5a8e45d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:37,088 [236] [DEBUG] [app] Ending request: urn:request:1ed879a1-091d-43c0-bb09-cca75535c3de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ed879a1-091d-43c0-bb09-cca75535c3de', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:18:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" blobuploadcleanupworker stdout | 2023-01-17 05:18:37,205 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:18:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:18:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:37.537407+00:00 (in 0.280376 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:18:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:37 UTC)" (scheduled at 2023-01-17 05:18:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:18:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:18:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 37, 257448), True, datetime.datetime(2023, 1, 17, 5, 18, 37, 257448), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:18:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:18:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:18:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:18:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:18:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:37.256652+00:00 (in 59.718901 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:18:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:37 UTC)" (scheduled at 2023-01-17 05:18:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:18:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:18:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:37 UTC)" executed successfully builder stdout | 2023-01-17 05:18:38,810 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:38,822 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:38,822 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:38,832 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:18:40,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:18:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:10.738568+00:00 (in 29.999443 seconds) buildlogsarchiver stdout | 2023-01-17 05:18:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:10 UTC)" (scheduled at 2023-01-17 05:18:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:18:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 18, 40, 739511), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:18:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:18:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:18:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:18:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:18:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:42.941536+00:00 (in 1.792807 seconds) namespacegcworker stdout | 2023-01-17 05:18:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:41 UTC)" (scheduled at 2023-01-17 05:18:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:18:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:18:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 41, 149208), True, datetime.datetime(2023, 1, 17, 5, 18, 41, 149208), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:18:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:18:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:18:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:18:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:18:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:41.148245+00:00 (in 58.206242 seconds) namespacegcworker stdout | 2023-01-17 05:18:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:42 UTC)" (scheduled at 2023-01-17 05:18:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:18:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:18:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:19:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:55.646354+00:00 (in 9.999479 seconds) notificationworker stdout | 2023-01-17 05:18:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:55 UTC)" (scheduled at 2023-01-17 05:18:45.646354+00:00) notificationworker stdout | 2023-01-17 05:18:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 45, 647325), True, datetime.datetime(2023, 1, 17, 5, 18, 45, 647325), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:18:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:18:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:18:49.537472+00:00 (in 2.600059 seconds) securityworker stdout | 2023-01-17 05:18:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:16 UTC)" (scheduled at 2023-01-17 05:18:46.936988+00:00) securityworker stdout | 2023-01-17 05:18:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:16 UTC)" executed successfully builder stdout | 2023-01-17 05:18:48,842 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:48,854 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:48,854 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:48,865 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:18:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:16.936988+00:00 (in 27.399048 seconds) securityworker stdout | 2023-01-17 05:18:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:19 UTC)" (scheduled at 2023-01-17 05:18:49.537472+00:00) securityworker stdout | 2023-01-17 05:18:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:18:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:18:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:18:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:18:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:18:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:18:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:18:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:18:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:18:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:18:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:18:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:18:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:18:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:18:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:18:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:21.348605+00:00 (in 29.999552 seconds) gcworker stdout | 2023-01-17 05:18:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:21 UTC)" (scheduled at 2023-01-17 05:18:51.348605+00:00) gcworker stdout | 2023-01-17 05:18:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723131349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:18:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:18:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:18:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:18:52,059 [236] [DEBUG] [app] Starting request: urn:request:9a883cae-99e9-494b-8d51-8fc3a8c1ec90 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:52,059 [235] [DEBUG] [app] Starting request: urn:request:e8c0edd9-b24c-4f53-b0f2-2a9ae8b4a79e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:18:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:18:52,064 [230] [DEBUG] [app] Starting request: urn:request:712c3ba7-b267-4d73-8949-54a4b59eb752 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:52,065 [241] [DEBUG] [app] Starting request: urn:request:d68738ef-44bd-49a3-8a29-ce2daded27ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:18:52,065 [230] [DEBUG] [app] Ending request: urn:request:712c3ba7-b267-4d73-8949-54a4b59eb752 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:712c3ba7-b267-4d73-8949-54a4b59eb752', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:18:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:18:52,065 [241] [DEBUG] [app] Ending request: urn:request:d68738ef-44bd-49a3-8a29-ce2daded27ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d68738ef-44bd-49a3-8a29-ce2daded27ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:18:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:18:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:18:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:18:52,070 [231] [DEBUG] [app] Starting request: urn:request:2050339c-471e-4f4b-9a51-ebea774fa07a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:52,070 [233] [DEBUG] [app] Starting request: urn:request:fe68e377-9245-4c80-a3ef-85efeeca56e0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:18:52,071 [231] [DEBUG] [app] Ending request: urn:request:2050339c-471e-4f4b-9a51-ebea774fa07a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2050339c-471e-4f4b-9a51-ebea774fa07a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:18:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:18:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:52,071 [233] [DEBUG] [app] Ending request: urn:request:fe68e377-9245-4c80-a3ef-85efeeca56e0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe68e377-9245-4c80-a3ef-85efeeca56e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:18:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:18:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:18:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:18:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:18:52,088 [236] [DEBUG] [app] Ending request: urn:request:9a883cae-99e9-494b-8d51-8fc3a8c1ec90 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a883cae-99e9-494b-8d51-8fc3a8c1ec90', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:52,088 [235] [DEBUG] [app] Ending request: urn:request:e8c0edd9-b24c-4f53-b0f2-2a9ae8b4a79e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e8c0edd9-b24c-4f53-b0f2-2a9ae8b4a79e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:18:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:18:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:18:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) notificationworker stdout | 2023-01-17 05:18:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:18:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:01.636362+00:00 (in 5.989560 seconds) notificationworker stdout | 2023-01-17 05:18:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:05 UTC)" (scheduled at 2023-01-17 05:18:55.646354+00:00) notificationworker stdout | 2023-01-17 05:18:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:18:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 18, 55, 647217), True, datetime.datetime(2023, 1, 17, 5, 18, 55, 647217), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:18:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:18:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:18:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:18:58,143 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:18:58,229 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:18:58,867 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:18:58,880 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:18:58,880 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:18:58,890 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:18:59,195 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:18:59,606 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:18:59,643 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:18:59,953 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:19:00,101 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:19:00,879 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:19:00,881 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:19:00,884 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:19:00,887 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:19:00,889 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:19:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:05.646354+00:00 (in 4.009515 seconds) notificationworker stdout | 2023-01-17 05:19:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:01 UTC)" (scheduled at 2023-01-17 05:19:01.636362+00:00) notificationworker stdout | 2023-01-17 05:19:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:19:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:19:03,194 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:19:03,522 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:19:04,609 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:19:04,612 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:19:04,869 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:19:04,873 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:19:04,876 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:19:04,881 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} repositorygcworker stdout | 2023-01-17 05:19:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:19:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:19.346120+00:00 (in 14.408879 seconds) repositorygcworker stdout | 2023-01-17 05:19:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:04 UTC)" (scheduled at 2023-01-17 05:19:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:19:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:19:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 4, 937659), True, datetime.datetime(2023, 1, 17, 5, 19, 4, 937659), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:19:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:19:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:19:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:19:05,149 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:19:05,151 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:19:05,161 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:19:05,164 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:19:05,166 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:19:05,168 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:19:05,178 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:19:05,180 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:19:05,182 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:19:05,434 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:19:05,643 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:19:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:15.646354+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 05:19:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:15 UTC)" (scheduled at 2023-01-17 05:19:05.646354+00:00) notificationworker stdout | 2023-01-17 05:19:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 5, 647196), True, datetime.datetime(2023, 1, 17, 5, 19, 5, 647196), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:15 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:19:05,870 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:19:05,877 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:19:06,411 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:19:06,971 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:19:07,001 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:19:07,059 [233] [DEBUG] [app] Starting request: urn:request:d8126f00-c0ea-49d4-b045-68a3675f6c22 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:07,059 [231] [DEBUG] [app] Starting request: urn:request:e619cfe4-38b0-48eb-870f-56b511ed9980 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:07,064 [243] [DEBUG] [app] Starting request: urn:request:ff4fe98b-69f6-45e9-ac2c-0f03bb33143b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:07,064 [230] [DEBUG] [app] Starting request: urn:request:e058bac0-71ad-4e49-abb5-33eef96c1636 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:07,065 [243] [DEBUG] [app] Ending request: urn:request:ff4fe98b-69f6-45e9-ac2c-0f03bb33143b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff4fe98b-69f6-45e9-ac2c-0f03bb33143b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:19:07,065 [230] [DEBUG] [app] Ending request: urn:request:e058bac0-71ad-4e49-abb5-33eef96c1636 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e058bac0-71ad-4e49-abb5-33eef96c1636', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:07,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:19:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:19:07,066 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:07,068 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:07,069 [235] [DEBUG] [app] Starting request: urn:request:6ab7f3bc-cbaa-4823-8fe9-5155c3c92805 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:07,070 [233] [DEBUG] [app] Starting request: urn:request:a654c645-a67f-4442-9a9b-a56fbf51fa7b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:07,070 [235] [DEBUG] [app] Ending request: urn:request:6ab7f3bc-cbaa-4823-8fe9-5155c3c92805 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6ab7f3bc-cbaa-4823-8fe9-5155c3c92805', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:07,070 [233] [DEBUG] [app] Ending request: urn:request:a654c645-a67f-4442-9a9b-a56fbf51fa7b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a654c645-a67f-4442-9a9b-a56fbf51fa7b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:19:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:07,087 [233] [DEBUG] [app] Ending request: urn:request:d8126f00-c0ea-49d4-b045-68a3675f6c22 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d8126f00-c0ea-49d4-b045-68a3675f6c22', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:07,087 [231] [DEBUG] [app] Ending request: urn:request:e619cfe4-38b0-48eb-870f-56b511ed9980 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e619cfe4-38b0-48eb-870f-56b511ed9980', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:07,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:07,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:19:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) blobuploadcleanupworker stdout | 2023-01-17 05:19:07,226 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:19:08,897 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:08,909 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:08,910 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:08,920 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:19:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:19:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:40.738568+00:00 (in 29.999505 seconds) buildlogsarchiver stdout | 2023-01-17 05:19:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:40 UTC)" (scheduled at 2023-01-17 05:19:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:19:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 19, 10, 739448), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:19:10,759 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:19:10,759 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:19:10,759 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:25.646354+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:19:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:25 UTC)" (scheduled at 2023-01-17 05:19:15.646354+00:00) notificationworker stdout | 2023-01-17 05:19:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 15, 647158), True, datetime.datetime(2023, 1, 17, 5, 19, 15, 647158), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:19:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:19.537472+00:00 (in 2.600012 seconds) securityworker stdout | 2023-01-17 05:19:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:46 UTC)" (scheduled at 2023-01-17 05:19:16.936988+00:00) securityworker stdout | 2023-01-17 05:19:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:46 UTC)" executed successfully builder stdout | 2023-01-17 05:19:18,929 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:18,942 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:18,942 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:18,952 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:19:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:19:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:04.936835+00:00 (in 45.590227 seconds) repositorygcworker stdout | 2023-01-17 05:19:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:19 UTC)" (scheduled at 2023-01-17 05:19:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:19:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:19:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:19:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:46.936988+00:00 (in 27.399032 seconds) securityworker stdout | 2023-01-17 05:19:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:49 UTC)" (scheduled at 2023-01-17 05:19:19.537472+00:00) securityworker stdout | 2023-01-17 05:19:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:19:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:19:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:19:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:19:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:19:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:19:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:19:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:19:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:51.348605+00:00 (in 29.999531 seconds) gcworker stdout | 2023-01-17 05:19:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:51 UTC)" (scheduled at 2023-01-17 05:19:21.348605+00:00) gcworker stdout | 2023-01-17 05:19:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723161349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:19:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:19:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:19:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:19:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:19:22,060 [235] [DEBUG] [app] Starting request: urn:request:3abc3a3d-3663-416f-be41-3b89083cdae7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:22,060 [233] [DEBUG] [app] Starting request: urn:request:403e42db-31d5-4816-9992-cdd53d096082 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:22,065 [241] [DEBUG] [app] Starting request: urn:request:dadfc848-592b-402b-823e-ca086d2c5970 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:22,066 [243] [DEBUG] [app] Starting request: urn:request:08e177eb-922f-4e77-9425-0c7361edfdd3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:22,066 [241] [DEBUG] [app] Ending request: urn:request:dadfc848-592b-402b-823e-ca086d2c5970 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dadfc848-592b-402b-823e-ca086d2c5970', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:19:22,066 [243] [DEBUG] [app] Ending request: urn:request:08e177eb-922f-4e77-9425-0c7361edfdd3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08e177eb-922f-4e77-9425-0c7361edfdd3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:19:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:19:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:22,070 [231] [DEBUG] [app] Starting request: urn:request:6113dff5-e83f-4ad5-adca-476e88dc3a31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:22,071 [233] [DEBUG] [app] Starting request: urn:request:26e0c80b-35cc-473d-b2cb-a76de440bf7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:22,071 [231] [DEBUG] [app] Ending request: urn:request:6113dff5-e83f-4ad5-adca-476e88dc3a31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6113dff5-e83f-4ad5-adca-476e88dc3a31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:22,071 [233] [DEBUG] [app] Ending request: urn:request:26e0c80b-35cc-473d-b2cb-a76de440bf7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26e0c80b-35cc-473d-b2cb-a76de440bf7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:22,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:22,088 [233] [DEBUG] [app] Ending request: urn:request:403e42db-31d5-4816-9992-cdd53d096082 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:403e42db-31d5-4816-9992-cdd53d096082', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:22,088 [235] [DEBUG] [app] Ending request: urn:request:3abc3a3d-3663-416f-be41-3b89083cdae7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3abc3a3d-3663-416f-be41-3b89083cdae7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:22,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:19:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) notificationworker stdout | 2023-01-17 05:19:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:35.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 05:19:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:35 UTC)" (scheduled at 2023-01-17 05:19:25.646354+00:00) notificationworker stdout | 2023-01-17 05:19:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 25, 647221), True, datetime.datetime(2023, 1, 17, 5, 19, 25, 647221), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:28,160 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:19:28,254 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:19:28,955 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue exportactionlogsworker stdout | 2023-01-17 05:19:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:19:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:35.853396+00:00 (in 6.889319 seconds) exportactionlogsworker stdout | 2023-01-17 05:19:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:28 UTC)" (scheduled at 2023-01-17 05:19:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:19:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:19:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:28 UTC)" executed successfully builder stdout | 2023-01-17 05:19:28,967 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:28,967 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:28,978 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:19:29,230 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:19:29,642 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:19:29,677 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:19:29,974 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:19:30,106 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:19:30,886 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:19:30,889 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:19:30,891 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:19:30,893 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:19:30,895 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:19:33,230 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:19:33,557 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:19:34,641 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:19:34,647 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:19:34,874 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:19:34,878 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:19:34,882 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:19:34,896 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:19:35,159 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:19:35,163 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:19:35,170 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:19:35,173 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:19:35,175 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:19:35,177 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:19:35,187 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:19:35,190 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:19:35,192 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:19:35,447 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:19:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:45.646354+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 05:19:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:45 UTC)" (scheduled at 2023-01-17 05:19:35.646354+00:00) notificationworker stdout | 2023-01-17 05:19:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 35, 647222), True, datetime.datetime(2023, 1, 17, 5, 19, 35, 647222), 0, 'notification/%', 50, 1, 0]) manifestbackfillworker stdout | 2023-01-17 05:19:35,649 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:19:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:19:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:19:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:28.963662+00:00 (in 53.109803 seconds) exportactionlogsworker stdout | 2023-01-17 05:19:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:35 UTC)" (scheduled at 2023-01-17 05:19:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:19:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:19:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 35, 854295), True, datetime.datetime(2023, 1, 17, 5, 19, 35, 854295), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:19:35,870 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:19:35,870 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:19:35,870 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:35 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:19:35,902 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:19:35,905 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:19:36,446 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:19:37,007 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:19:37,009 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:19:37,060 [236] [DEBUG] [app] Starting request: urn:request:92d03a08-af41-4021-ba7f-96e63ecc1c5d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:37,060 [231] [DEBUG] [app] Starting request: urn:request:c59ca052-a4d6-4c80-ab3b-93592febaaea (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:37,065 [230] [DEBUG] [app] Starting request: urn:request:b9839d0a-a563-4ee3-81e8-e702c1a473e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:37,066 [243] [DEBUG] [app] Starting request: urn:request:81e247bd-1d42-4efb-9595-275c2afb86ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:37,066 [230] [DEBUG] [app] Ending request: urn:request:b9839d0a-a563-4ee3-81e8-e702c1a473e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9839d0a-a563-4ee3-81e8-e702c1a473e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:37,066 [243] [DEBUG] [app] Ending request: urn:request:81e247bd-1d42-4efb-9595-275c2afb86ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81e247bd-1d42-4efb-9595-275c2afb86ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:19:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:19:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:37,071 [236] [DEBUG] [app] Starting request: urn:request:11c2c434-bd10-46bb-a941-db799aff2fee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:37,071 [236] [DEBUG] [app] Ending request: urn:request:11c2c434-bd10-46bb-a941-db799aff2fee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11c2c434-bd10-46bb-a941-db799aff2fee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:37,072 [235] [DEBUG] [app] Starting request: urn:request:78dcef65-69bf-4a47-98bb-ee9b71f24ef3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:37,072 [235] [DEBUG] [app] Ending request: urn:request:78dcef65-69bf-4a47-98bb-ee9b71f24ef3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78dcef65-69bf-4a47-98bb-ee9b71f24ef3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:37,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:37,089 [236] [DEBUG] [app] Ending request: urn:request:92d03a08-af41-4021-ba7f-96e63ecc1c5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:92d03a08-af41-4021-ba7f-96e63ecc1c5d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:37,089 [231] [DEBUG] [app] Ending request: urn:request:c59ca052-a4d6-4c80-ab3b-93592febaaea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c59ca052-a4d6-4c80-ab3b-93592febaaea', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:37,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:37,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:19:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:19:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:19:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:37.537407+00:00 (in 0.280352 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:19:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:37 UTC)" (scheduled at 2023-01-17 05:19:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:19:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:19:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 37, 257491), True, datetime.datetime(2023, 1, 17, 5, 19, 37, 257491), 0, 'secscanv4/%', 50, 1, 0]) blobuploadcleanupworker stdout | 2023-01-17 05:19:37,262 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:19:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:19:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:19:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:19:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:19:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:37.256652+00:00 (in 59.718827 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:19:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:37 UTC)" (scheduled at 2023-01-17 05:19:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:19:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:19:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:37 UTC)" executed successfully builder stdout | 2023-01-17 05:19:38,988 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:39,000 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:39,000 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:39,011 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:19:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:19:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:10.738568+00:00 (in 29.999512 seconds) buildlogsarchiver stdout | 2023-01-17 05:19:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:10 UTC)" (scheduled at 2023-01-17 05:19:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:19:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 19, 40, 739460), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:19:40,756 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:19:40,756 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:19:40,756 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:19:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:19:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:42.941536+00:00 (in 1.792805 seconds) namespacegcworker stdout | 2023-01-17 05:19:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:41 UTC)" (scheduled at 2023-01-17 05:19:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:19:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:19:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 41, 149131), True, datetime.datetime(2023, 1, 17, 5, 19, 41, 149131), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:19:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:19:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:19:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:19:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:19:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:41.148245+00:00 (in 58.206200 seconds) namespacegcworker stdout | 2023-01-17 05:19:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:42 UTC)" (scheduled at 2023-01-17 05:19:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:19:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:19:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:20:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:55.646354+00:00 (in 9.999575 seconds) notificationworker stdout | 2023-01-17 05:19:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:55 UTC)" (scheduled at 2023-01-17 05:19:45.646354+00:00) notificationworker stdout | 2023-01-17 05:19:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 45, 647157), True, datetime.datetime(2023, 1, 17, 5, 19, 45, 647157), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:19:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:19:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:19:49.537472+00:00 (in 2.600005 seconds) securityworker stdout | 2023-01-17 05:19:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:16 UTC)" (scheduled at 2023-01-17 05:19:46.936988+00:00) securityworker stdout | 2023-01-17 05:19:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:16 UTC)" executed successfully builder stdout | 2023-01-17 05:19:49,013 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:49,025 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:49,025 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:49,036 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:19:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:16.936988+00:00 (in 27.399039 seconds) securityworker stdout | 2023-01-17 05:19:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:19 UTC)" (scheduled at 2023-01-17 05:19:49.537472+00:00) securityworker stdout | 2023-01-17 05:19:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:19:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:19:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:19:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:19:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:19:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:19:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:19:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:19:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:19:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:19:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:19:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:19:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:19:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:19:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:19:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:21.348605+00:00 (in 29.999511 seconds) gcworker stdout | 2023-01-17 05:19:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:21 UTC)" (scheduled at 2023-01-17 05:19:51.348605+00:00) gcworker stdout | 2023-01-17 05:19:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723191349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:19:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:19:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:19:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:19:52,060 [231] [DEBUG] [app] Starting request: urn:request:53c786e8-04b6-4d40-8804-028271e7af88 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:52,060 [233] [DEBUG] [app] Starting request: urn:request:6b894218-275f-4d1e-b34a-0ad649aafba1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:19:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:52,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:19:52,065 [230] [DEBUG] [app] Starting request: urn:request:dc21b781-3747-4543-abfc-f083f3ef1bd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,065 [242] [DEBUG] [app] Starting request: urn:request:9d371ad9-c3bf-4823-a110-f4a84a3c6d09 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,066 [230] [DEBUG] [app] Ending request: urn:request:dc21b781-3747-4543-abfc-f083f3ef1bd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc21b781-3747-4543-abfc-f083f3ef1bd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,066 [242] [DEBUG] [app] Ending request: urn:request:9d371ad9-c3bf-4823-a110-f4a84a3c6d09 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d371ad9-c3bf-4823-a110-f4a84a3c6d09', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:19:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:19:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:19:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:19:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:19:52,071 [236] [DEBUG] [app] Starting request: urn:request:e27059c4-44d3-40ce-bebe-5b30f18b766c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:52,071 [231] [DEBUG] [app] Starting request: urn:request:1fde7f89-95eb-4da0-bde4-e60944a64dea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:19:52,071 [236] [DEBUG] [app] Ending request: urn:request:e27059c4-44d3-40ce-bebe-5b30f18b766c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e27059c4-44d3-40ce-bebe-5b30f18b766c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:19:52,071 [231] [DEBUG] [app] Ending request: urn:request:1fde7f89-95eb-4da0-bde4-e60944a64dea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1fde7f89-95eb-4da0-bde4-e60944a64dea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:19:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:52,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:19:52,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:19:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:19:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:52,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:19:52,088 [231] [DEBUG] [app] Ending request: urn:request:53c786e8-04b6-4d40-8804-028271e7af88 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:53c786e8-04b6-4d40-8804-028271e7af88', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:52,088 [233] [DEBUG] [app] Ending request: urn:request:6b894218-275f-4d1e-b34a-0ad649aafba1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b894218-275f-4d1e-b34a-0ad649aafba1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:19:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:19:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:19:52,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:19:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:19:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:01.636362+00:00 (in 5.989553 seconds) notificationworker stdout | 2023-01-17 05:19:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:05 UTC)" (scheduled at 2023-01-17 05:19:55.646354+00:00) notificationworker stdout | 2023-01-17 05:19:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:19:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 19, 55, 647209), True, datetime.datetime(2023, 1, 17, 5, 19, 55, 647209), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:19:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:19:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:19:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:19:58,183 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:19:58,286 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:19:59,046 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:19:59,058 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:19:59,058 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:19:59,069 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:19:59,243 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:19:59,660 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:19:59,712 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:20:00,006 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:20:00,141 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:20:00,893 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:20:00,896 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:20:00,898 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:20:00,900 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:20:00,902 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:20:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:05.646354+00:00 (in 4.009489 seconds) notificationworker stdout | 2023-01-17 05:20:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:01 UTC)" (scheduled at 2023-01-17 05:20:01.636362+00:00) notificationworker stdout | 2023-01-17 05:20:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:20:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:20:03,260 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:20:03,585 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:20:04,647 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:20:04,681 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:20:04,880 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:20:04,885 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:20:04,887 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:20:04,928 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} repositorygcworker stdout | 2023-01-17 05:20:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:20:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:19.346120+00:00 (in 14.408935 seconds) repositorygcworker stdout | 2023-01-17 05:20:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:04 UTC)" (scheduled at 2023-01-17 05:20:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:20:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:20:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 4, 937557), True, datetime.datetime(2023, 1, 17, 5, 20, 4, 937557), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:20:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:20:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:20:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:04 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:20:05,170 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:20:05,173 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:20:05,178 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:20:05,180 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:20:05,183 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:20:05,186 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:20:05,193 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:20:05,198 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:20:05,200 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:20:05,483 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:20:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:15.646354+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 05:20:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:15 UTC)" (scheduled at 2023-01-17 05:20:05.646354+00:00) notificationworker stdout | 2023-01-17 05:20:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 5, 647168), True, datetime.datetime(2023, 1, 17, 5, 20, 5, 647168), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:15 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:20:05,670 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositoryactioncounter stdout | 2023-01-17 05:20:05,910 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:20:05,928 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:20:06,462 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:20:07,043 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:20:07,045 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:20:07,059 [235] [DEBUG] [app] Starting request: urn:request:85c033dd-2602-4e9a-9d6f-57308f1b2279 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:07,059 [231] [DEBUG] [app] Starting request: urn:request:86fec4db-59f5-48be-be49-15de6164cd9f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:07,064 [230] [DEBUG] [app] Starting request: urn:request:6bcf9cb3-ec10-4ac2-b722-38354c9910e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:07,064 [242] [DEBUG] [app] Starting request: urn:request:211e4367-9334-4bfd-8541-334375d2e96f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:07,065 [230] [DEBUG] [app] Ending request: urn:request:6bcf9cb3-ec10-4ac2-b722-38354c9910e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6bcf9cb3-ec10-4ac2-b722-38354c9910e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:20:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:20:07,065 [242] [DEBUG] [app] Ending request: urn:request:211e4367-9334-4bfd-8541-334375d2e96f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:211e4367-9334-4bfd-8541-334375d2e96f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:20:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:20:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:07,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:07,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:07,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:07,069 [233] [DEBUG] [app] Starting request: urn:request:9cae1cad-c082-45cd-b014-f2e8218f2d23 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:07,069 [231] [DEBUG] [app] Starting request: urn:request:a9f8043e-d6e6-4026-ba74-fab09829d90a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:07,070 [233] [DEBUG] [app] Ending request: urn:request:9cae1cad-c082-45cd-b014-f2e8218f2d23 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9cae1cad-c082-45cd-b014-f2e8218f2d23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:07,070 [231] [DEBUG] [app] Ending request: urn:request:a9f8043e-d6e6-4026-ba74-fab09829d90a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9f8043e-d6e6-4026-ba74-fab09829d90a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:20:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:07,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:07,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:07,086 [231] [DEBUG] [app] Ending request: urn:request:86fec4db-59f5-48be-be49-15de6164cd9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:86fec4db-59f5-48be-be49-15de6164cd9f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:07,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:07,086 [235] [DEBUG] [app] Ending request: urn:request:85c033dd-2602-4e9a-9d6f-57308f1b2279 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:85c033dd-2602-4e9a-9d6f-57308f1b2279', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:20:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" blobuploadcleanupworker stdout | 2023-01-17 05:20:07,287 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:20:09,078 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:09,090 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:09,090 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:09,101 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:20:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:20:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:40.738568+00:00 (in 29.999536 seconds) buildlogsarchiver stdout | 2023-01-17 05:20:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:40 UTC)" (scheduled at 2023-01-17 05:20:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:20:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 20, 10, 739433), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:20:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:20:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:20:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:25.646354+00:00 (in 9.999587 seconds) notificationworker stdout | 2023-01-17 05:20:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:25 UTC)" (scheduled at 2023-01-17 05:20:15.646354+00:00) notificationworker stdout | 2023-01-17 05:20:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 15, 647171), True, datetime.datetime(2023, 1, 17, 5, 20, 15, 647171), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:20:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:19.537472+00:00 (in 2.600018 seconds) securityworker stdout | 2023-01-17 05:20:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:46 UTC)" (scheduled at 2023-01-17 05:20:16.936988+00:00) securityworker stdout | 2023-01-17 05:20:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:46 UTC)" executed successfully builder stdout | 2023-01-17 05:20:19,111 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:19,123 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:19,123 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:19,134 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:20:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:20:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:04.936835+00:00 (in 45.590277 seconds) repositorygcworker stdout | 2023-01-17 05:20:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:19 UTC)" (scheduled at 2023-01-17 05:20:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:20:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:20:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:20:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:46.936988+00:00 (in 27.399086 seconds) securityworker stdout | 2023-01-17 05:20:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:49 UTC)" (scheduled at 2023-01-17 05:20:19.537472+00:00) securityworker stdout | 2023-01-17 05:20:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:20:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:20:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:20:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:20:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:20:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:20:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:20:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:20:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:51.348605+00:00 (in 29.999498 seconds) gcworker stdout | 2023-01-17 05:20:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:51 UTC)" (scheduled at 2023-01-17 05:20:21.348605+00:00) gcworker stdout | 2023-01-17 05:20:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723221349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:20:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:20:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:20:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:20:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:20:22,060 [235] [DEBUG] [app] Starting request: urn:request:656b6565-5703-4a3e-bd63-9c687436cb8e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:22,060 [231] [DEBUG] [app] Starting request: urn:request:a0512546-3fba-45de-90dd-abaa9945facd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:22,065 [230] [DEBUG] [app] Starting request: urn:request:e91d2536-d221-4791-8eed-17cae1c8eeb9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:22,065 [242] [DEBUG] [app] Starting request: urn:request:5b798ee2-d467-47a4-90f2-94aa5fa28d70 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:22,065 [230] [DEBUG] [app] Ending request: urn:request:e91d2536-d221-4791-8eed-17cae1c8eeb9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e91d2536-d221-4791-8eed-17cae1c8eeb9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:20:22,066 [242] [DEBUG] [app] Ending request: urn:request:5b798ee2-d467-47a4-90f2-94aa5fa28d70 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5b798ee2-d467-47a4-90f2-94aa5fa28d70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:20:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:20:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:22,070 [233] [DEBUG] [app] Starting request: urn:request:421d6f79-e0db-4312-9b29-4b2367e9ee62 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:22,070 [231] [DEBUG] [app] Starting request: urn:request:159b3b2f-cf2c-4e80-9f2b-a04d355314db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:22,070 [233] [DEBUG] [app] Ending request: urn:request:421d6f79-e0db-4312-9b29-4b2367e9ee62 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:421d6f79-e0db-4312-9b29-4b2367e9ee62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:22,070 [231] [DEBUG] [app] Ending request: urn:request:159b3b2f-cf2c-4e80-9f2b-a04d355314db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:159b3b2f-cf2c-4e80-9f2b-a04d355314db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:22,087 [231] [DEBUG] [app] Ending request: urn:request:a0512546-3fba-45de-90dd-abaa9945facd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0512546-3fba-45de-90dd-abaa9945facd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:22,088 [235] [DEBUG] [app] Ending request: urn:request:656b6565-5703-4a3e-bd63-9c687436cb8e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:656b6565-5703-4a3e-bd63-9c687436cb8e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:20:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:20:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:35.646354+00:00 (in 9.999565 seconds) notificationworker stdout | 2023-01-17 05:20:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:35 UTC)" (scheduled at 2023-01-17 05:20:25.646354+00:00) notificationworker stdout | 2023-01-17 05:20:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 25, 647183), True, datetime.datetime(2023, 1, 17, 5, 20, 25, 647183), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:28,218 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:20:28,311 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:20:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:20:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:35.853396+00:00 (in 6.889262 seconds) exportactionlogsworker stdout | 2023-01-17 05:20:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:28 UTC)" (scheduled at 2023-01-17 05:20:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:20:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:20:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:28 UTC)" executed successfully builder stdout | 2023-01-17 05:20:29,144 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:29,155 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:29,156 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:29,166 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:20:29,279 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:20:29,687 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:20:29,740 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:20:30,020 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:20:30,176 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:20:30,900 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:20:30,902 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:20:30,904 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:20:30,908 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:20:30,910 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:20:33,279 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:20:33,615 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:20:34,675 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:20:34,686 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:20:34,886 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:20:34,891 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:20:34,894 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:20:34,963 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:20:35,180 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:20:35,182 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:20:35,187 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:20:35,190 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:20:35,192 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:20:35,195 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:20:35,200 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:20:35,206 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:20:35,210 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:20:35,518 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:20:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:45.646354+00:00 (in 9.999546 seconds) notificationworker stdout | 2023-01-17 05:20:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:45 UTC)" (scheduled at 2023-01-17 05:20:35.646354+00:00) notificationworker stdout | 2023-01-17 05:20:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 35, 647214), True, datetime.datetime(2023, 1, 17, 5, 20, 35, 647214), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:45 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:20:35,692 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} exportactionlogsworker stdout | 2023-01-17 05:20:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:20:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:28.963662+00:00 (in 53.109782 seconds) exportactionlogsworker stdout | 2023-01-17 05:20:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:35 UTC)" (scheduled at 2023-01-17 05:20:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:20:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:20:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 35, 854269), True, datetime.datetime(2023, 1, 17, 5, 20, 35, 854269), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:20:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:20:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:20:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:35 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:20:35,945 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:20:35,947 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:20:36,486 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:20:37,060 [236] [DEBUG] [app] Starting request: urn:request:2baeafa2-e455-4661-b044-7687d5539050 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:37,060 [235] [DEBUG] [app] Starting request: urn:request:7e991fa1-8299-428c-8c64-1161bfae4bc5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost exportactionlogsworker stdout | 2023-01-17 05:20:37,062 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} gunicorn-web stdout | 2023-01-17 05:20:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:37,064 [243] [DEBUG] [app] Starting request: urn:request:296f8319-5cfe-4ff2-b39d-7be9577dd8bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:37,065 [243] [DEBUG] [app] Ending request: urn:request:296f8319-5cfe-4ff2-b39d-7be9577dd8bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:296f8319-5cfe-4ff2-b39d-7be9577dd8bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:37,065 [230] [DEBUG] [app] Starting request: urn:request:a36914f0-1768-449e-bbe3-458be5792500 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:20:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:20:37,065 [230] [DEBUG] [app] Ending request: urn:request:a36914f0-1768-449e-bbe3-458be5792500 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a36914f0-1768-449e-bbe3-458be5792500', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:20:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:37,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:37,069 [231] [DEBUG] [app] Starting request: urn:request:d422e8ba-9d1a-48c5-b914-61e283d0a1ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:37,070 [231] [DEBUG] [app] Ending request: urn:request:d422e8ba-9d1a-48c5-b914-61e283d0a1ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d422e8ba-9d1a-48c5-b914-61e283d0a1ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:20:37,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:37,071 [231] [DEBUG] [app] Starting request: urn:request:08f7bf4d-8192-47d2-b726-4e6eec36da68 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:37,071 [231] [DEBUG] [app] Ending request: urn:request:08f7bf4d-8192-47d2-b726-4e6eec36da68 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:08f7bf4d-8192-47d2-b726-4e6eec36da68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql teamsyncworker stdout | 2023-01-17 05:20:37,077 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} gunicorn-web stdout | 2023-01-17 05:20:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:37,087 [236] [DEBUG] [app] Ending request: urn:request:2baeafa2-e455-4661-b044-7687d5539050 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2baeafa2-e455-4661-b044-7687d5539050', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:37,087 [235] [DEBUG] [app] Ending request: urn:request:7e991fa1-8299-428c-8c64-1161bfae4bc5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e991fa1-8299-428c-8c64-1161bfae4bc5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:20:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:20:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:20:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:37.537407+00:00 (in 0.280334 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:20:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:37 UTC)" (scheduled at 2023-01-17 05:20:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:20:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:20:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 37, 257505), True, datetime.datetime(2023, 1, 17, 5, 20, 37, 257505), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:20:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:20:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:20:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:20:37,322 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:20:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:20:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:23.537048+00:00 (in 45.999233 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:20:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:37 UTC)" (scheduled at 2023-01-17 05:20:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:20:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:20:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:37 UTC)" executed successfully builder stdout | 2023-01-17 05:20:39,174 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:39,186 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:39,187 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:39,198 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:20:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:20:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:10.738568+00:00 (in 29.999524 seconds) buildlogsarchiver stdout | 2023-01-17 05:20:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:10 UTC)" (scheduled at 2023-01-17 05:20:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:20:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 20, 40, 739465), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:20:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:20:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:20:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:20:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:20:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:42.941536+00:00 (in 1.792844 seconds) namespacegcworker stdout | 2023-01-17 05:20:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:41 UTC)" (scheduled at 2023-01-17 05:20:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:20:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:20:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 41, 149071), True, datetime.datetime(2023, 1, 17, 5, 20, 41, 149071), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:20:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:20:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:20:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:20:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:20:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:41.148245+00:00 (in 58.206233 seconds) namespacegcworker stdout | 2023-01-17 05:20:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:42 UTC)" (scheduled at 2023-01-17 05:20:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:20:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:20:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:21:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:55.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:20:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:55 UTC)" (scheduled at 2023-01-17 05:20:45.646354+00:00) notificationworker stdout | 2023-01-17 05:20:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 45, 647208), True, datetime.datetime(2023, 1, 17, 5, 20, 45, 647208), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:20:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:20:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:20:49.537472+00:00 (in 2.600027 seconds) securityworker stdout | 2023-01-17 05:20:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:16 UTC)" (scheduled at 2023-01-17 05:20:46.936988+00:00) securityworker stdout | 2023-01-17 05:20:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:16 UTC)" executed successfully builder stdout | 2023-01-17 05:20:49,208 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:49,220 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:49,220 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:49,231 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:20:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:16.936988+00:00 (in 27.399003 seconds) securityworker stdout | 2023-01-17 05:20:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:19 UTC)" (scheduled at 2023-01-17 05:20:49.537472+00:00) securityworker stdout | 2023-01-17 05:20:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:20:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:20:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:20:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:20:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:20:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:20:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:20:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:20:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:20:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:20:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:20:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:20:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:20:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:20:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:20:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:21.348605+00:00 (in 29.999509 seconds) gcworker stdout | 2023-01-17 05:20:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:21 UTC)" (scheduled at 2023-01-17 05:20:51.348605+00:00) gcworker stdout | 2023-01-17 05:20:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723251349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:20:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:20:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:20:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:20:52,060 [231] [DEBUG] [app] Starting request: urn:request:cb31ab3a-e4d3-4f43-aba1-e70462d2aa5a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:52,060 [233] [DEBUG] [app] Starting request: urn:request:8f0c8aea-4e60-4f2d-8e26-98dac78078dc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:20:52,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:52,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:20:52,066 [241] [DEBUG] [app] Starting request: urn:request:20e6680a-7548-4f43-affb-4c57d4947c9a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,066 [230] [DEBUG] [app] Starting request: urn:request:86ce7b87-f01c-4890-b68c-63cdb7dc62e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,066 [241] [DEBUG] [app] Ending request: urn:request:20e6680a-7548-4f43-affb-4c57d4947c9a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20e6680a-7548-4f43-affb-4c57d4947c9a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,067 [230] [DEBUG] [app] Ending request: urn:request:86ce7b87-f01c-4890-b68c-63cdb7dc62e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86ce7b87-f01c-4890-b68c-63cdb7dc62e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:20:52,067 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:20:52,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:52,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:20:52,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:52,069 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,069 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:20:52,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:52,071 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:20:52,072 [235] [DEBUG] [app] Starting request: urn:request:5abe94e0-4120-4f14-b3e3-1573f196572e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:52,072 [236] [DEBUG] [app] Starting request: urn:request:63b021ca-b23a-4ae9-9623-af679540b4ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:20:52,072 [235] [DEBUG] [app] Ending request: urn:request:5abe94e0-4120-4f14-b3e3-1573f196572e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5abe94e0-4120-4f14-b3e3-1573f196572e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:52,072 [236] [DEBUG] [app] Ending request: urn:request:63b021ca-b23a-4ae9-9623-af679540b4ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:63b021ca-b23a-4ae9-9623-af679540b4ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:20:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:20:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:52,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:20:52,073 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:20:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:52,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:20:52,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:52,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:20:52,089 [231] [DEBUG] [app] Ending request: urn:request:cb31ab3a-e4d3-4f43-aba1-e70462d2aa5a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb31ab3a-e4d3-4f43-aba1-e70462d2aa5a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:52,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:20:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:20:52,098 [233] [DEBUG] [app] Ending request: urn:request:8f0c8aea-4e60-4f2d-8e26-98dac78078dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f0c8aea-4e60-4f2d-8e26-98dac78078dc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:20:52,098 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:20:52,098 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.039 47 0.039) notificationworker stdout | 2023-01-17 05:20:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:20:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:01.636362+00:00 (in 5.989521 seconds) notificationworker stdout | 2023-01-17 05:20:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:05 UTC)" (scheduled at 2023-01-17 05:20:55.646354+00:00) notificationworker stdout | 2023-01-17 05:20:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:20:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 20, 55, 647265), True, datetime.datetime(2023, 1, 17, 5, 20, 55, 647265), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:20:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:20:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:20:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:20:58,254 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:20:58,339 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} builder stdout | 2023-01-17 05:20:59,234 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:20:59,246 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:20:59,246 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:20:59,257 [60] [DEBUG] [data.database] Disconnecting from database. queuecleanupworker stdout | 2023-01-17 05:20:59,285 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:20:59,716 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:20:59,763 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:21:00,055 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:21:00,211 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:21:00,907 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:21:00,909 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:21:00,911 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:21:00,914 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:21:00,916 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:21:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:05.646354+00:00 (in 4.009502 seconds) notificationworker stdout | 2023-01-17 05:21:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:01 UTC)" (scheduled at 2023-01-17 05:21:01.636362+00:00) notificationworker stdout | 2023-01-17 05:21:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:21:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:21:03,315 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:21:03,623 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:21:04,712 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:21:04,716 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:21:04,892 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:21:04,898 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:21:04,900 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:21:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:21:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:17.345601+00:00 (in 12.408337 seconds) repositorygcworker stdout | 2023-01-17 05:21:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:04 UTC)" (scheduled at 2023-01-17 05:21:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:21:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:21:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 4, 937686), True, datetime.datetime(2023, 1, 17, 5, 21, 4, 937686), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:21:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:21:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:21:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:21:04,968 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:21:05,189 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:21:05,191 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:21:05,195 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:21:05,198 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:21:05,200 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:21:05,202 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:21:05,207 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:21:05,212 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:21:05,220 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:21:05,543 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:21:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:15.646354+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:21:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:15 UTC)" (scheduled at 2023-01-17 05:21:05.646354+00:00) notificationworker stdout | 2023-01-17 05:21:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 5, 647217), True, datetime.datetime(2023, 1, 17, 5, 21, 5, 647217), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:15 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:21:05,715 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:21:05,966 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:21:05,982 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:21:06,518 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:21:07,059 [231] [DEBUG] [app] Starting request: urn:request:e058fe31-7ce8-4d35-b73c-13734b03676e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:07,059 [235] [DEBUG] [app] Starting request: urn:request:45d6a756-c2b8-41a0-afbd-f929abf9e842 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:07,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,060 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:07,064 [230] [DEBUG] [app] Starting request: urn:request:9e1a2d41-ab4f-482d-9843-b01421161e82 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:07,065 [243] [DEBUG] [app] Starting request: urn:request:54f20487-3906-4e2e-addb-4deb4c5ef1a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:07,065 [230] [DEBUG] [app] Ending request: urn:request:9e1a2d41-ab4f-482d-9843-b01421161e82 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e1a2d41-ab4f-482d-9843-b01421161e82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:21:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:21:07,065 [243] [DEBUG] [app] Ending request: urn:request:54f20487-3906-4e2e-addb-4deb4c5ef1a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54f20487-3906-4e2e-addb-4deb4c5ef1a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:21:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:21:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:07,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:07,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:07,069 [236] [DEBUG] [app] Starting request: urn:request:a91d3b59-2859-4579-a96b-66efc28e96ea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:07,070 [236] [DEBUG] [app] Ending request: urn:request:a91d3b59-2859-4579-a96b-66efc28e96ea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a91d3b59-2859-4579-a96b-66efc28e96ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:07,070 [233] [DEBUG] [app] Starting request: urn:request:c2a78fd8-736c-4250-b200-689f2352a9a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:07,070 [233] [DEBUG] [app] Ending request: urn:request:c2a78fd8-736c-4250-b200-689f2352a9a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2a78fd8-736c-4250-b200-689f2352a9a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:07,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:07,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:07,087 [231] [DEBUG] [app] Ending request: urn:request:e058fe31-7ce8-4d35-b73c-13734b03676e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e058fe31-7ce8-4d35-b73c-13734b03676e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:07,087 [235] [DEBUG] [app] Ending request: urn:request:45d6a756-c2b8-41a0-afbd-f929abf9e842 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:45d6a756-c2b8-41a0-afbd-f929abf9e842', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:07,087 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:21:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:21:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" teamsyncworker stdout | 2023-01-17 05:21:07,090 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:21:07,093 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:21:07,334 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:21:09,267 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:09,279 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:09,279 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:09,290 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:21:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:21:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:40.738568+00:00 (in 29.999517 seconds) buildlogsarchiver stdout | 2023-01-17 05:21:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:40 UTC)" (scheduled at 2023-01-17 05:21:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:21:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 21, 10, 739454), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:21:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:21:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:21:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:25.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:21:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:25 UTC)" (scheduled at 2023-01-17 05:21:15.646354+00:00) notificationworker stdout | 2023-01-17 05:21:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 15, 647181), True, datetime.datetime(2023, 1, 17, 5, 21, 15, 647181), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:21:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:19.537472+00:00 (in 2.599983 seconds) securityworker stdout | 2023-01-17 05:21:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:46 UTC)" (scheduled at 2023-01-17 05:21:16.936988+00:00) securityworker stdout | 2023-01-17 05:21:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:21:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:21:17,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:21:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:19.346120+00:00 (in 1.999996 seconds) repositorygcworker stdout | 2023-01-17 05:21:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:17 UTC)" (scheduled at 2023-01-17 05:21:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:21:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 21, 17, 346554), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:21:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 17, 346554), True, datetime.datetime(2023, 1, 17, 5, 21, 17, 346554), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:21:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 17, 346554), True, datetime.datetime(2023, 1, 17, 5, 21, 17, 346554), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 21, 17, 346554), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:21:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:21:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:17 UTC)" executed successfully builder stdout | 2023-01-17 05:21:19,296 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:19,308 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:19,308 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:19,319 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:21:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:21:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:04.936835+00:00 (in 45.590236 seconds) repositorygcworker stdout | 2023-01-17 05:21:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:19 UTC)" (scheduled at 2023-01-17 05:21:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:21:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:21:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:21:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:46.936988+00:00 (in 27.399044 seconds) securityworker stdout | 2023-01-17 05:21:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:49 UTC)" (scheduled at 2023-01-17 05:21:19.537472+00:00) securityworker stdout | 2023-01-17 05:21:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:21:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:21:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:21:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:19,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:19,556 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:19,556 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:21:19,556 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:21:19,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:21:19,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:21:19,558 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:21:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:21:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:51.348605+00:00 (in 29.999516 seconds) gcworker stdout | 2023-01-17 05:21:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:51 UTC)" (scheduled at 2023-01-17 05:21:21.348605+00:00) gcworker stdout | 2023-01-17 05:21:21,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:21:21,360 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723281359, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:21:21,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:21:21,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:21:21,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:21:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:21:22,060 [233] [DEBUG] [app] Starting request: urn:request:a30782f0-3a5f-4b40-bd70-2c67888025fa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:22,061 [231] [DEBUG] [app] Starting request: urn:request:2e97bed1-3b04-4ad6-a6fc-40091d5d66bc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:22,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:22,066 [242] [DEBUG] [app] Starting request: urn:request:03d58d4e-d4db-4bd4-adb4-46aadb8c2071 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:22,066 [230] [DEBUG] [app] Starting request: urn:request:12338040-ac3b-477b-8430-f7e6285ba81b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:22,066 [242] [DEBUG] [app] Ending request: urn:request:03d58d4e-d4db-4bd4-adb4-46aadb8c2071 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03d58d4e-d4db-4bd4-adb4-46aadb8c2071', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:21:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:21:22,066 [230] [DEBUG] [app] Ending request: urn:request:12338040-ac3b-477b-8430-f7e6285ba81b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:12338040-ac3b-477b-8430-f7e6285ba81b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:21:22,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:21:22,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,069 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:22,071 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:22,071 [236] [DEBUG] [app] Starting request: urn:request:bd0095b8-53c4-45e7-b0b5-61d1459ea63d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:22,071 [236] [DEBUG] [app] Ending request: urn:request:bd0095b8-53c4-45e7-b0b5-61d1459ea63d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bd0095b8-53c4-45e7-b0b5-61d1459ea63d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:22,071 [231] [DEBUG] [app] Starting request: urn:request:9b986c50-7579-4f32-9bee-453ef7f3518c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:22,072 [231] [DEBUG] [app] Ending request: urn:request:9b986c50-7579-4f32-9bee-453ef7f3518c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b986c50-7579-4f32-9bee-453ef7f3518c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:22,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:22,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:22,081 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:22,089 [233] [DEBUG] [app] Ending request: urn:request:a30782f0-3a5f-4b40-bd70-2c67888025fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a30782f0-3a5f-4b40-bd70-2c67888025fa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:22,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:22,090 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-web stdout | 2023-01-17 05:21:22,090 [231] [DEBUG] [app] Ending request: urn:request:2e97bed1-3b04-4ad6-a6fc-40091d5d66bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e97bed1-3b04-4ad6-a6fc-40091d5d66bc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:22,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:22,091 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) securityscanningnotificationworker stdout | 2023-01-17 05:21:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:21:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:37.256652+00:00 (in 13.719173 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:21:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:23 UTC)" (scheduled at 2023-01-17 05:21:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:21:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 21, 23, 537839), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:21:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 23, 537839), True, datetime.datetime(2023, 1, 17, 5, 21, 23, 537839), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:21:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 23, 537839), True, datetime.datetime(2023, 1, 17, 5, 21, 23, 537839), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 21, 23, 537839), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:21:23,555 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:21:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:35.646354+00:00 (in 9.999587 seconds) notificationworker stdout | 2023-01-17 05:21:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:35 UTC)" (scheduled at 2023-01-17 05:21:25.646354+00:00) notificationworker stdout | 2023-01-17 05:21:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 25, 647175), True, datetime.datetime(2023, 1, 17, 5, 21, 25, 647175), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:28,270 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:21:28,374 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:21:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:21:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:35.853396+00:00 (in 6.889276 seconds) exportactionlogsworker stdout | 2023-01-17 05:21:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:28 UTC)" (scheduled at 2023-01-17 05:21:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:21:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:21:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:21:29,312 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:21:29,321 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:29,333 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:29,333 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:29,343 [60] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2023-01-17 05:21:29,744 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:21:29,798 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:21:30,086 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:21:30,219 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:21:30,914 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:21:30,917 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:21:30,919 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:21:30,922 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:21:30,924 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:21:32,744 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 05:21:32,745 [64] [INFO] [apscheduler.executors.default] Running job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 05:21:32 UTC)" (scheduled at 2023-01-17 05:21:32.744591+00:00) expiredappspecifictokenworker stdout | 2023-01-17 05:21:32,745 [64] [DEBUG] [__main__] Garbage collecting expired app specific tokens with window: 1 day, 0:00:00 expiredappspecifictokenworker stdout | 2023-01-17 05:21:32,746 [64] [DEBUG] [peewee] ('DELETE FROM "appspecificauthtoken" WHERE ("appspecificauthtoken"."expiration" < %s)', [datetime.datetime(2023, 1, 16, 5, 21, 32, 745722)]) expiredappspecifictokenworker stdout | 2023-01-17 05:21:32,746 [64] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:32.744591+00:00 (in 3599.999640 seconds) expiredappspecifictokenworker stdout | 2023-01-17 05:21:32,756 [64] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:21:32,756 [64] [INFO] [apscheduler.executors.default] Job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 06:21:32 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:21:33,347 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:21:33,659 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} storagereplication stdout | 2023-01-17 05:21:34,747 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:21:34,751 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:21:34,899 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:21:34,905 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:21:34,907 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:21:34,990 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:21:35,197 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:21:35,200 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:21:35,203 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:21:35,205 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:21:35,207 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:21:35,210 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:21:35,213 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:21:35,217 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:21:35,229 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:21:35,578 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:21:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:45.646354+00:00 (in 9.999582 seconds) notificationworker stdout | 2023-01-17 05:21:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:45 UTC)" (scheduled at 2023-01-17 05:21:35.646354+00:00) notificationworker stdout | 2023-01-17 05:21:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 35, 647165), True, datetime.datetime(2023, 1, 17, 5, 21, 35, 647165), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:45 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:21:35,729 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} exportactionlogsworker stdout | 2023-01-17 05:21:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:21:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:54.963413+00:00 (in 19.109497 seconds) exportactionlogsworker stdout | 2023-01-17 05:21:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:35 UTC)" (scheduled at 2023-01-17 05:21:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:21:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:21:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 35, 854415), True, datetime.datetime(2023, 1, 17, 5, 21, 35, 854415), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:21:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:21:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:21:35,866 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:35 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:21:36,001 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:21:36,016 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:21:36,546 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:21:37,059 [231] [DEBUG] [app] Starting request: urn:request:cc28b7a8-9ac7-4e99-97be-388c372ad934 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:37,060 [236] [DEBUG] [app] Starting request: urn:request:49724015-20a9-4693-b52a-cb941f8e16c8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:37,065 [240] [DEBUG] [app] Starting request: urn:request:2883e130-0497-4599-8c4a-1efd72d4b760 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:37,065 [240] [DEBUG] [app] Ending request: urn:request:2883e130-0497-4599-8c4a-1efd72d4b760 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2883e130-0497-4599-8c4a-1efd72d4b760', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:21:37,065 [239] [DEBUG] [app] Starting request: urn:request:94087d3c-7636-46e4-a72f-388b5ebe6c12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:37,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:21:37,065 [239] [DEBUG] [app] Ending request: urn:request:94087d3c-7636-46e4-a72f-388b5ebe6c12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:94087d3c-7636-46e4-a72f-388b5ebe6c12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:21:37,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:21:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:37,071 [235] [DEBUG] [app] Starting request: urn:request:a58c6d78-173d-4027-bca6-1da7dbb7614a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:37,071 [235] [DEBUG] [app] Ending request: urn:request:a58c6d78-173d-4027-bca6-1da7dbb7614a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a58c6d78-173d-4027-bca6-1da7dbb7614a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:37,071 [233] [DEBUG] [app] Starting request: urn:request:4fcd4117-d4f4-479d-959c-d124dc714075 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:21:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:37,071 [233] [DEBUG] [app] Ending request: urn:request:4fcd4117-d4f4-479d-959c-d124dc714075 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4fcd4117-d4f4-479d-959c-d124dc714075', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:37,088 [236] [DEBUG] [app] Ending request: urn:request:49724015-20a9-4693-b52a-cb941f8e16c8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:49724015-20a9-4693-b52a-cb941f8e16c8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:37,088 [231] [DEBUG] [app] Ending request: urn:request:cc28b7a8-9ac7-4e99-97be-388c372ad934 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc28b7a8-9ac7-4e99-97be-388c372ad934', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:21:37,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-web stdout | 2023-01-17 05:21:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 05:21:37,114 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:21:37,125 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} securityscanningnotificationworker stdout | 2023-01-17 05:21:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:21:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:37.537407+00:00 (in 0.280296 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:21:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:37 UTC)" (scheduled at 2023-01-17 05:21:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:21:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:21:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 37, 257546), True, datetime.datetime(2023, 1, 17, 5, 21, 37, 257546), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:21:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:21:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:21:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:21:37,369 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:21:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:21:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:37.256652+00:00 (in 59.718809 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:21:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:37 UTC)" (scheduled at 2023-01-17 05:21:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:21:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:21:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:37 UTC)" executed successfully builder stdout | 2023-01-17 05:21:39,353 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:39,364 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:39,364 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:39,375 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:21:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:21:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:10.738568+00:00 (in 29.999541 seconds) buildlogsarchiver stdout | 2023-01-17 05:21:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:10 UTC)" (scheduled at 2023-01-17 05:21:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:21:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 21, 40, 739427), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:21:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:21:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:21:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:21:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:21:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:42.941536+00:00 (in 1.792785 seconds) namespacegcworker stdout | 2023-01-17 05:21:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:41 UTC)" (scheduled at 2023-01-17 05:21:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:21:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:21:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 41, 149148), True, datetime.datetime(2023, 1, 17, 5, 21, 41, 149148), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:21:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:21:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:21:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:21:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:21:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:06.941185+00:00 (in 23.999175 seconds) namespacegcworker stdout | 2023-01-17 05:21:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:42 UTC)" (scheduled at 2023-01-17 05:21:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:21:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:21:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:22:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:55.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:21:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:55 UTC)" (scheduled at 2023-01-17 05:21:45.646354+00:00) notificationworker stdout | 2023-01-17 05:21:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 45, 647208), True, datetime.datetime(2023, 1, 17, 5, 21, 45, 647208), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:21:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:21:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:21:49.537472+00:00 (in 2.599992 seconds) securityworker stdout | 2023-01-17 05:21:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:16 UTC)" (scheduled at 2023-01-17 05:21:46.936988+00:00) securityworker stdout | 2023-01-17 05:21:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:21:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:16 UTC)" executed successfully builder stdout | 2023-01-17 05:21:49,385 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:49,397 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:49,397 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:49,408 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:21:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:16.936988+00:00 (in 27.399061 seconds) securityworker stdout | 2023-01-17 05:21:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:19 UTC)" (scheduled at 2023-01-17 05:21:49.537472+00:00) securityworker stdout | 2023-01-17 05:21:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:21:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:49,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:21:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:21:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:21:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:21:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:21:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:21:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:21:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:21:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:21:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:21:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:21:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:21:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:21:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:21:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:21.348605+00:00 (in 29.999501 seconds) gcworker stdout | 2023-01-17 05:21:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:21 UTC)" (scheduled at 2023-01-17 05:21:51.348605+00:00) gcworker stdout | 2023-01-17 05:21:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723311349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:21:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:21:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:21:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:21:52,060 [236] [DEBUG] [app] Starting request: urn:request:78348e7b-2e7b-4649-9895-8473401bfa9c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:52,060 [235] [DEBUG] [app] Starting request: urn:request:ff62aaad-47fa-427a-a0c0-534be5c1391c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:21:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:21:52,064 [242] [DEBUG] [app] Starting request: urn:request:be5e83b3-17a3-4e0e-8a71-3fa70d2ce9c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:52,065 [243] [DEBUG] [app] Starting request: urn:request:1ad88527-26b8-4e5c-9af6-867fe01eca25 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:21:52,065 [242] [DEBUG] [app] Ending request: urn:request:be5e83b3-17a3-4e0e-8a71-3fa70d2ce9c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be5e83b3-17a3-4e0e-8a71-3fa70d2ce9c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:21:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:21:52,065 [243] [DEBUG] [app] Ending request: urn:request:1ad88527-26b8-4e5c-9af6-867fe01eca25 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1ad88527-26b8-4e5c-9af6-867fe01eca25', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:21:52,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:21:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:21:52,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:21:52,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:21:52,070 [231] [DEBUG] [app] Starting request: urn:request:e8f745ed-5221-4f9b-9b79-186c7ab9bc55 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:52,070 [233] [DEBUG] [app] Starting request: urn:request:a7e337bf-3381-452e-9dc3-0264e2c9feac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:21:52,070 [231] [DEBUG] [app] Ending request: urn:request:e8f745ed-5221-4f9b-9b79-186c7ab9bc55 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e8f745ed-5221-4f9b-9b79-186c7ab9bc55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:21:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:21:52,071 [233] [DEBUG] [app] Ending request: urn:request:a7e337bf-3381-452e-9dc3-0264e2c9feac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7e337bf-3381-452e-9dc3-0264e2c9feac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:21:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:21:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:21:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:21:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:21:52,087 [235] [DEBUG] [app] Ending request: urn:request:ff62aaad-47fa-427a-a0c0-534be5c1391c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ff62aaad-47fa-427a-a0c0-534be5c1391c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:52,087 [236] [DEBUG] [app] Ending request: urn:request:78348e7b-2e7b-4649-9895-8473401bfa9c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78348e7b-2e7b-4649-9895-8473401bfa9c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:21:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:21:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:21:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 05:21:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:21:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:28.963662+00:00 (in 33.999752 seconds) exportactionlogsworker stdout | 2023-01-17 05:21:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:54 UTC)" (scheduled at 2023-01-17 05:21:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:21:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 21, 54, 964327), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:21:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 54, 964327), True, datetime.datetime(2023, 1, 17, 5, 21, 54, 964327), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:21:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 21, 54, 964327), True, datetime.datetime(2023, 1, 17, 5, 21, 54, 964327), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 21, 54, 964327), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:21:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:21:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:26:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:21:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:01.636362+00:00 (in 5.989559 seconds) notificationworker stdout | 2023-01-17 05:21:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:05 UTC)" (scheduled at 2023-01-17 05:21:55.646354+00:00) notificationworker stdout | 2023-01-17 05:21:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:21:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 21, 55, 647205), True, datetime.datetime(2023, 1, 17, 5, 21, 55, 647205), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:21:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:21:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:21:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:21:58,297 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:21:58,405 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:21:59,345 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:21:59,418 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:21:59,430 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:21:59,430 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:21:59,440 [60] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2023-01-17 05:21:59,779 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:21:59,833 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:22:00,121 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:22:00,254 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:22:00,921 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:22:00,924 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:22:00,927 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:22:00,929 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:22:00,931 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:22:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:05.646354+00:00 (in 4.009497 seconds) notificationworker stdout | 2023-01-17 05:22:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:01 UTC)" (scheduled at 2023-01-17 05:22:01.636362+00:00) notificationworker stdout | 2023-01-17 05:22:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:22:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:22:03,375 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:22:03,685 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:22:04,763 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:22:04,775 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-secscan stdout | 2023-01-17 05:22:04,907 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:22:04,911 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:22:04,913 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:22:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:22:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:19.346120+00:00 (in 14.408871 seconds) repositorygcworker stdout | 2023-01-17 05:22:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:04 UTC)" (scheduled at 2023-01-17 05:22:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:22:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:22:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 4, 937631), True, datetime.datetime(2023, 1, 17, 5, 22, 4, 937631), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:22:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:22:04,948 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:22:04,948 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:22:05,025 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:22:05,205 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:22:05,208 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:22:05,210 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:22:05,213 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:22:05,216 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:22:05,218 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:22:05,223 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:22:05,228 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:22:05,236 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:22:05,584 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:22:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:15.646354+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 05:22:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:15 UTC)" (scheduled at 2023-01-17 05:22:05.646354+00:00) notificationworker stdout | 2023-01-17 05:22:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 5, 647209), True, datetime.datetime(2023, 1, 17, 5, 22, 5, 647209), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:15 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:22:05,764 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:22:06,036 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:22:06,048 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:22:06,581 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} namespacegcworker stdout | 2023-01-17 05:22:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:22:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:41.148245+00:00 (in 34.206600 seconds) namespacegcworker stdout | 2023-01-17 05:22:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:27:06 UTC)" (scheduled at 2023-01-17 05:22:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:22:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 22, 6, 942036), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:22:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 22, 6, 942036), True, datetime.datetime(2023, 1, 17, 5, 22, 6, 942036), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:22:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 22, 6, 942036), True, datetime.datetime(2023, 1, 17, 5, 22, 6, 942036), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 22, 6, 942036), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:22:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:22:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:27:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:22:07,059 [231] [DEBUG] [app] Starting request: urn:request:71291319-5c77-405d-9ded-73bff7257003 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:07,060 [233] [DEBUG] [app] Starting request: urn:request:83611352-eacd-4a0a-9ffe-c82035301c2b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:07,064 [243] [DEBUG] [app] Starting request: urn:request:07c9a039-ba50-48ee-98d9-a164a359609f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:07,065 [243] [DEBUG] [app] Ending request: urn:request:07c9a039-ba50-48ee-98d9-a164a359609f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07c9a039-ba50-48ee-98d9-a164a359609f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:07,065 [239] [DEBUG] [app] Starting request: urn:request:abb3267a-98c3-4527-930a-cb59b4ab5c16 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:22:07,065 [239] [DEBUG] [app] Ending request: urn:request:abb3267a-98c3-4527-930a-cb59b4ab5c16 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abb3267a-98c3-4527-930a-cb59b4ab5c16', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:07,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:22:07,065 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:07,066 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:07,068 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:07,069 [235] [DEBUG] [app] Starting request: urn:request:58186068-e9f1-40ce-ba2a-0d80bb0a4848 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:07,069 [236] [DEBUG] [app] Starting request: urn:request:aed745b4-4ac1-4ac2-95f4-6fab05c54fa5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:07,070 [235] [DEBUG] [app] Ending request: urn:request:58186068-e9f1-40ce-ba2a-0d80bb0a4848 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58186068-e9f1-40ce-ba2a-0d80bb0a4848', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:07,070 [236] [DEBUG] [app] Ending request: urn:request:aed745b4-4ac1-4ac2-95f4-6fab05c54fa5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aed745b4-4ac1-4ac2-95f4-6fab05c54fa5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:07,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:07,070 [233] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:22:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:07,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:07,086 [233] [DEBUG] [app] Ending request: urn:request:83611352-eacd-4a0a-9ffe-c82035301c2b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83611352-eacd-4a0a-9ffe-c82035301c2b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:07,086 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:07,086 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:22:07,087 [231] [DEBUG] [app] Ending request: urn:request:71291319-5c77-405d-9ded-73bff7257003 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71291319-5c77-405d-9ded-73bff7257003', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.028) gunicorn-web stdout | 2023-01-17 05:22:07,087 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:22:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 05:22:07,150 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:22:07,160 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:22:07,391 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:22:09,450 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:09,462 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:09,462 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:09,472 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:22:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:22:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:40.738568+00:00 (in 29.999575 seconds) buildlogsarchiver stdout | 2023-01-17 05:22:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:40 UTC)" (scheduled at 2023-01-17 05:22:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:22:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 22, 10, 739434), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:22:10,759 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:22:10,759 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:22:10,759 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:25.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:22:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:25 UTC)" (scheduled at 2023-01-17 05:22:15.646354+00:00) notificationworker stdout | 2023-01-17 05:22:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 15, 647224), True, datetime.datetime(2023, 1, 17, 5, 22, 15, 647224), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:22:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:19.537472+00:00 (in 2.600013 seconds) securityworker stdout | 2023-01-17 05:22:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:46 UTC)" (scheduled at 2023-01-17 05:22:16.936988+00:00) securityworker stdout | 2023-01-17 05:22:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:22:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:22:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:04.936835+00:00 (in 45.590260 seconds) repositorygcworker stdout | 2023-01-17 05:22:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:19 UTC)" (scheduled at 2023-01-17 05:22:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:22:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:22:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:19 UTC)" executed successfully builder stdout | 2023-01-17 05:22:19,482 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:19,494 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:19,494 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:19,505 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:46.936988+00:00 (in 27.399028 seconds) securityworker stdout | 2023-01-17 05:22:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:49 UTC)" (scheduled at 2023-01-17 05:22:19.537472+00:00) securityworker stdout | 2023-01-17 05:22:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:22:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:22:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:22:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:22:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:22:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:22:19,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:22:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:22:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:51.348605+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 05:22:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:51 UTC)" (scheduled at 2023-01-17 05:22:21.348605+00:00) gcworker stdout | 2023-01-17 05:22:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723341349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:22:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:22:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:22:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:22:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:22:22,060 [236] [DEBUG] [app] Starting request: urn:request:8b7f9753-89d0-4250-822b-8c3e016c582c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:22,060 [231] [DEBUG] [app] Starting request: urn:request:735af7eb-f6e9-43eb-a14b-9c0fc3b26a24 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:22,065 [230] [DEBUG] [app] Starting request: urn:request:3f5ae3dd-fe5b-4a91-871c-aedb8eacd4ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:22,066 [242] [DEBUG] [app] Starting request: urn:request:37f5de32-4780-4585-89da-21ea026c8166 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:22,066 [230] [DEBUG] [app] Ending request: urn:request:3f5ae3dd-fe5b-4a91-871c-aedb8eacd4ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f5ae3dd-fe5b-4a91-871c-aedb8eacd4ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:22:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:22:22,066 [242] [DEBUG] [app] Ending request: urn:request:37f5de32-4780-4585-89da-21ea026c8166 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:37f5de32-4780-4585-89da-21ea026c8166', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:22:22,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:22,071 [235] [DEBUG] [app] Starting request: urn:request:0b18e20f-b3a8-4546-ae84-c9753447deeb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:22,071 [233] [DEBUG] [app] Starting request: urn:request:066c6858-5175-4435-be49-6910b85ba8f8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:22,071 [235] [DEBUG] [app] Ending request: urn:request:0b18e20f-b3a8-4546-ae84-c9753447deeb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b18e20f-b3a8-4546-ae84-c9753447deeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:22,071 [233] [DEBUG] [app] Ending request: urn:request:066c6858-5175-4435-be49-6910b85ba8f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:066c6858-5175-4435-be49-6910b85ba8f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:22:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:22,088 [231] [DEBUG] [app] Ending request: urn:request:735af7eb-f6e9-43eb-a14b-9c0fc3b26a24 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:735af7eb-f6e9-43eb-a14b-9c0fc3b26a24', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:22,088 [236] [DEBUG] [app] Ending request: urn:request:8b7f9753-89d0-4250-822b-8c3e016c582c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b7f9753-89d0-4250-822b-8c3e016c582c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:22:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:22:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:35.646354+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:22:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:35 UTC)" (scheduled at 2023-01-17 05:22:25.646354+00:00) notificationworker stdout | 2023-01-17 05:22:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 25, 647252), True, datetime.datetime(2023, 1, 17, 5, 22, 25, 647252), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:28,332 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:22:28,440 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:22:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:22:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:35.853396+00:00 (in 6.889248 seconds) exportactionlogsworker stdout | 2023-01-17 05:22:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:28 UTC)" (scheduled at 2023-01-17 05:22:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:22:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:22:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:22:29,353 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:22:29,515 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:29,527 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:29,527 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:29,538 [60] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2023-01-17 05:22:29,815 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:22:29,848 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:22:30,149 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:22:30,289 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:22:30,929 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:22:30,932 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:22:30,934 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:22:30,936 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:22:30,939 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:22:33,405 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:22:33,707 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:22:34,775 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:22:34,800 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-secscan stdout | 2023-01-17 05:22:34,913 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:22:34,917 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:22:34,919 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:22:35,032 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:22:35,214 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:22:35,217 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:22:35,219 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:22:35,223 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:22:35,225 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:22:35,228 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:22:35,233 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:22:35,237 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:22:35,244 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:22:35,606 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:22:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:45.646354+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 05:22:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:45 UTC)" (scheduled at 2023-01-17 05:22:35.646354+00:00) notificationworker stdout | 2023-01-17 05:22:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 35, 647171), True, datetime.datetime(2023, 1, 17, 5, 22, 35, 647171), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:45 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:22:35,773 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} exportactionlogsworker stdout | 2023-01-17 05:22:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:22:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:28.963662+00:00 (in 53.109745 seconds) exportactionlogsworker stdout | 2023-01-17 05:22:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:35 UTC)" (scheduled at 2023-01-17 05:22:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:22:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:22:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 35, 854372), True, datetime.datetime(2023, 1, 17, 5, 22, 35, 854372), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:22:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:22:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:22:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:35 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:22:36,071 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:22:36,083 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:22:36,617 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:22:37,060 [233] [DEBUG] [app] Starting request: urn:request:b80b6fec-d1f8-484a-a20a-42cbf33837bd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:37,060 [235] [DEBUG] [app] Starting request: urn:request:a838ac20-05b7-4d39-9a07-bc3786a4f019 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:37,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:37,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:37,065 [241] [DEBUG] [app] Starting request: urn:request:ec02a495-e905-46dd-8763-96febcbd0f3e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,066 [239] [DEBUG] [app] Starting request: urn:request:398e59ab-20d1-4535-b994-11ed32b3ed04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,066 [241] [DEBUG] [app] Ending request: urn:request:ec02a495-e905-46dd-8763-96febcbd0f3e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec02a495-e905-46dd-8763-96febcbd0f3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,066 [239] [DEBUG] [app] Ending request: urn:request:398e59ab-20d1-4535-b994-11ed32b3ed04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:398e59ab-20d1-4535-b994-11ed32b3ed04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:22:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:22:37,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:37,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:22:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:37,071 [236] [DEBUG] [app] Starting request: urn:request:452ee695-e497-4023-af3a-cdbae66b96b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:37,071 [236] [DEBUG] [app] Ending request: urn:request:452ee695-e497-4023-af3a-cdbae66b96b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:452ee695-e497-4023-af3a-cdbae66b96b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:37,071 [233] [DEBUG] [app] Starting request: urn:request:d90d4492-0af1-49da-8f3a-cf8713f15357 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:37,072 [233] [DEBUG] [app] Ending request: urn:request:d90d4492-0af1-49da-8f3a-cf8713f15357 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d90d4492-0af1-49da-8f3a-cf8713f15357', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:22:37,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:37,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:37,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:37,089 [233] [DEBUG] [app] Ending request: urn:request:b80b6fec-d1f8-484a-a20a-42cbf33837bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b80b6fec-d1f8-484a-a20a-42cbf33837bd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:37,089 [235] [DEBUG] [app] Ending request: urn:request:a838ac20-05b7-4d39-9a07-bc3786a4f019 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a838ac20-05b7-4d39-9a07-bc3786a4f019', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:37,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:37,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:37,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:22:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) exportactionlogsworker stdout | 2023-01-17 05:22:37,186 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:22:37,195 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} securityscanningnotificationworker stdout | 2023-01-17 05:22:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:22:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:37.537407+00:00 (in 0.280325 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:22:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:37 UTC)" (scheduled at 2023-01-17 05:22:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:22:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:22:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 37, 257525), True, datetime.datetime(2023, 1, 17, 5, 22, 37, 257525), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:22:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:22:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:22:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:22:37,427 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:22:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:22:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:37.256652+00:00 (in 59.718802 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:22:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:37 UTC)" (scheduled at 2023-01-17 05:22:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:22:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:22:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:37 UTC)" executed successfully builder stdout | 2023-01-17 05:22:39,544 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:39,556 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:39,556 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:39,567 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:22:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:22:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:10.738568+00:00 (in 29.999543 seconds) buildlogsarchiver stdout | 2023-01-17 05:22:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:10 UTC)" (scheduled at 2023-01-17 05:22:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:22:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 22, 40, 739447), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:22:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:22:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:22:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:22:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:22:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:42.941536+00:00 (in 1.792787 seconds) namespacegcworker stdout | 2023-01-17 05:22:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:41 UTC)" (scheduled at 2023-01-17 05:22:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:22:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:22:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 41, 149147), True, datetime.datetime(2023, 1, 17, 5, 22, 41, 149147), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:22:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:22:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:22:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:22:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:22:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:41.148245+00:00 (in 58.206239 seconds) namespacegcworker stdout | 2023-01-17 05:22:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:42 UTC)" (scheduled at 2023-01-17 05:22:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:22:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:22:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:23:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:55.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 05:22:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:55 UTC)" (scheduled at 2023-01-17 05:22:45.646354+00:00) notificationworker stdout | 2023-01-17 05:22:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 45, 647228), True, datetime.datetime(2023, 1, 17, 5, 22, 45, 647228), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:22:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:22:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:22:49.537472+00:00 (in 2.600004 seconds) securityworker stdout | 2023-01-17 05:22:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:16 UTC)" (scheduled at 2023-01-17 05:22:46.936988+00:00) securityworker stdout | 2023-01-17 05:22:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:22:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:22:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:16.936988+00:00 (in 27.399083 seconds) securityworker stdout | 2023-01-17 05:22:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:19 UTC)" (scheduled at 2023-01-17 05:22:49.537472+00:00) securityworker stdout | 2023-01-17 05:22:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:22:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:22:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:49,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:22:49,540 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:22:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:22:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:22:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:22:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:22:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:22:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:22:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:22:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:22:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:22:49,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:19 UTC)" executed successfully builder stdout | 2023-01-17 05:22:49,577 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:49,588 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:49,588 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:49,599 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:22:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:22:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:21.348605+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 05:22:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:21 UTC)" (scheduled at 2023-01-17 05:22:51.348605+00:00) gcworker stdout | 2023-01-17 05:22:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723371349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:22:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:22:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:22:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:22:52,061 [235] [DEBUG] [app] Starting request: urn:request:bee506d9-162b-4fda-a8f4-45ea5692a282 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:52,061 [236] [DEBUG] [app] Starting request: urn:request:bbd0527b-eeb4-4a94-8abe-77a4b5bba78f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:22:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:52,066 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:22:52,066 [230] [DEBUG] [app] Starting request: urn:request:1c014356-3ecb-4c29-9ce6-de032f93031d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:22:52,066 [230] [DEBUG] [app] Ending request: urn:request:1c014356-3ecb-4c29-9ce6-de032f93031d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c014356-3ecb-4c29-9ce6-de032f93031d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:22:52,066 [242] [DEBUG] [app] Starting request: urn:request:0589d240-f1d3-414a-bf2e-0f85e47e0675 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:22:52,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:22:52,067 [242] [DEBUG] [app] Ending request: urn:request:0589d240-f1d3-414a-bf2e-0f85e47e0675 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0589d240-f1d3-414a-bf2e-0f85e47e0675', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:52,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:22:52,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:22:52,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:52,069 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:22:52,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:52,072 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:22:52,072 [236] [DEBUG] [app] Starting request: urn:request:781a57dd-2654-4996-9247-daaa54f8b3e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:22:52,072 [236] [DEBUG] [app] Ending request: urn:request:781a57dd-2654-4996-9247-daaa54f8b3e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:781a57dd-2654-4996-9247-daaa54f8b3e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:52,073 [231] [DEBUG] [app] Starting request: urn:request:efe896b0-57e6-471a-98c2-77024d27bfd6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:22:52,073 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:22:52,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:52,073 [231] [DEBUG] [app] Ending request: urn:request:efe896b0-57e6-471a-98c2-77024d27bfd6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:efe896b0-57e6-471a-98c2-77024d27bfd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:22:52,073 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:22:52,074 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:22:52,074 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:22:52,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:52,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:22:52,089 [236] [DEBUG] [app] Ending request: urn:request:bbd0527b-eeb4-4a94-8abe-77a4b5bba78f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bbd0527b-eeb4-4a94-8abe-77a4b5bba78f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:22:52,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:22:52,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:22:52,089 [235] [DEBUG] [app] Ending request: urn:request:bee506d9-162b-4fda-a8f4-45ea5692a282 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bee506d9-162b-4fda-a8f4-45ea5692a282', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:22:52,089 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:22:52,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:22:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:22:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:01.636362+00:00 (in 5.989523 seconds) notificationworker stdout | 2023-01-17 05:22:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:05 UTC)" (scheduled at 2023-01-17 05:22:55.646354+00:00) notificationworker stdout | 2023-01-17 05:22:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:22:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 22, 55, 647263), True, datetime.datetime(2023, 1, 17, 5, 22, 55, 647263), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:22:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:22:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:22:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:22:58,345 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:22:58,454 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:22:59,378 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:22:59,606 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:22:59,618 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:22:59,618 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:22:59,628 [60] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2023-01-17 05:22:59,849 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:22:59,870 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:23:00,179 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:23:00,324 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:23:00,935 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:23:00,938 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:23:00,941 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:23:00,943 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:23:00,945 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:23:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:05.646354+00:00 (in 4.009484 seconds) notificationworker stdout | 2023-01-17 05:23:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:01 UTC)" (scheduled at 2023-01-17 05:23:01.636362+00:00) notificationworker stdout | 2023-01-17 05:23:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:23:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:23:03,441 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:23:03,731 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:23:04,790 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:23:04,821 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-secscan stdout | 2023-01-17 05:23:04,920 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:23:04,924 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:23:04,926 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:23:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:23:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:19.346120+00:00 (in 14.408852 seconds) repositorygcworker stdout | 2023-01-17 05:23:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:04 UTC)" (scheduled at 2023-01-17 05:23:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:23:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:23:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 4, 937675), True, datetime.datetime(2023, 1, 17, 5, 23, 4, 937675), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:23:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:23:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:23:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:23:05,067 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:23:05,221 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:23:05,223 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:23:05,226 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:23:05,233 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:23:05,235 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:23:05,238 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:23:05,241 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:23:05,246 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:23:05,250 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} chunkcleanupworker stdout | 2023-01-17 05:23:05,615 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:23:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:15.646354+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:23:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:15 UTC)" (scheduled at 2023-01-17 05:23:05.646354+00:00) notificationworker stdout | 2023-01-17 05:23:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 5, 647183), True, datetime.datetime(2023, 1, 17, 5, 23, 5, 647183), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:15 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:23:05,790 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:23:06,083 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:23:06,118 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:23:06,653 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:23:07,060 [236] [DEBUG] [app] Starting request: urn:request:651a4f90-76de-496e-8e32-62f5dba7845a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:07,060 [235] [DEBUG] [app] Starting request: urn:request:c46c2403-b2b4-469d-9c29-a4d8c447d803 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:07,065 [230] [DEBUG] [app] Starting request: urn:request:e0ab4c00-9725-4aaa-bf35-16007bd954f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,065 [241] [DEBUG] [app] Starting request: urn:request:fd5338ab-1e4c-4572-91af-68cf8bdf8fbe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,065 [230] [DEBUG] [app] Ending request: urn:request:e0ab4c00-9725-4aaa-bf35-16007bd954f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e0ab4c00-9725-4aaa-bf35-16007bd954f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,065 [241] [DEBUG] [app] Ending request: urn:request:fd5338ab-1e4c-4572-91af-68cf8bdf8fbe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd5338ab-1e4c-4572-91af-68cf8bdf8fbe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:23:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:23:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:07,069 [235] [DEBUG] [app] Starting request: urn:request:8838c90d-b08f-45f8-841e-f07192f4b455 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:07,070 [235] [DEBUG] [app] Ending request: urn:request:8838c90d-b08f-45f8-841e-f07192f4b455 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8838c90d-b08f-45f8-841e-f07192f4b455', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:07,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:07,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:07,071 [233] [DEBUG] [app] Starting request: urn:request:24efed62-38ff-43a6-b8d8-4b74212b1a5b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:07,072 [233] [DEBUG] [app] Ending request: urn:request:24efed62-38ff-43a6-b8d8-4b74212b1a5b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:24efed62-38ff-43a6-b8d8-4b74212b1a5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:23:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:07,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:07,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:07,088 [235] [DEBUG] [app] Ending request: urn:request:c46c2403-b2b4-469d-9c29-a4d8c447d803 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c46c2403-b2b4-469d-9c29-a4d8c447d803', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:23:07,096 [236] [DEBUG] [app] Ending request: urn:request:651a4f90-76de-496e-8e32-62f5dba7845a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:651a4f90-76de-496e-8e32-62f5dba7845a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:07,096 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:07,096 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.038) exportactionlogsworker stdout | 2023-01-17 05:23:07,221 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:23:07,224 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:23:07,443 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:23:09,639 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:09,651 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:09,651 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:09,661 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:23:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:23:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:40.738568+00:00 (in 29.999549 seconds) buildlogsarchiver stdout | 2023-01-17 05:23:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:40 UTC)" (scheduled at 2023-01-17 05:23:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:23:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 23, 10, 739414), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:23:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:23:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:23:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:25.646354+00:00 (in 9.999591 seconds) notificationworker stdout | 2023-01-17 05:23:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:25 UTC)" (scheduled at 2023-01-17 05:23:15.646354+00:00) notificationworker stdout | 2023-01-17 05:23:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 15, 647193), True, datetime.datetime(2023, 1, 17, 5, 23, 15, 647193), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:23:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:19.537472+00:00 (in 2.600019 seconds) securityworker stdout | 2023-01-17 05:23:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:46 UTC)" (scheduled at 2023-01-17 05:23:16.936988+00:00) securityworker stdout | 2023-01-17 05:23:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:23:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:23:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:04.936835+00:00 (in 45.590272 seconds) repositorygcworker stdout | 2023-01-17 05:23:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:19 UTC)" (scheduled at 2023-01-17 05:23:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:23:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:23:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:23:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:46.936988+00:00 (in 27.399080 seconds) securityworker stdout | 2023-01-17 05:23:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:49 UTC)" (scheduled at 2023-01-17 05:23:19.537472+00:00) securityworker stdout | 2023-01-17 05:23:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:23:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:23:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:23:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:23:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:23:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:23:19,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:49 UTC)" executed successfully builder stdout | 2023-01-17 05:23:19,671 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:19,683 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:19,683 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:19,694 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:23:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:23:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:51.348605+00:00 (in 29.999532 seconds) gcworker stdout | 2023-01-17 05:23:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:51 UTC)" (scheduled at 2023-01-17 05:23:21.348605+00:00) gcworker stdout | 2023-01-17 05:23:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723401349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:23:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:23:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:23:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:23:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:23:22,059 [236] [DEBUG] [app] Starting request: urn:request:69bc925e-ac0b-48d4-bdd8-fcfa8216668d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:22,059 [231] [DEBUG] [app] Starting request: urn:request:77c54656-c65e-40b9-a218-8dd964f9843a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:22,060 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:22,064 [230] [DEBUG] [app] Starting request: urn:request:01fe1bac-3031-4cb9-aa83-741f5bcc134e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:22,064 [240] [DEBUG] [app] Starting request: urn:request:36313c89-347e-4684-b594-7035d80dec20 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:22,064 [230] [DEBUG] [app] Ending request: urn:request:01fe1bac-3031-4cb9-aa83-741f5bcc134e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01fe1bac-3031-4cb9-aa83-741f5bcc134e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:22,065 [240] [DEBUG] [app] Ending request: urn:request:36313c89-347e-4684-b594-7035d80dec20 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36313c89-347e-4684-b594-7035d80dec20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:23:22,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:23:22,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:22,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:22,069 [236] [DEBUG] [app] Starting request: urn:request:ca07f502-c602-4895-a070-33e95f1968e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:22,069 [233] [DEBUG] [app] Starting request: urn:request:cacae224-2d41-4e95-b362-43231702556a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:22,069 [236] [DEBUG] [app] Ending request: urn:request:ca07f502-c602-4895-a070-33e95f1968e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ca07f502-c602-4895-a070-33e95f1968e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:22,069 [233] [DEBUG] [app] Ending request: urn:request:cacae224-2d41-4e95-b362-43231702556a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cacae224-2d41-4e95-b362-43231702556a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:22,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:22,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:22,086 [236] [DEBUG] [app] Ending request: urn:request:69bc925e-ac0b-48d4-bdd8-fcfa8216668d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69bc925e-ac0b-48d4-bdd8-fcfa8216668d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:22,086 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:22,086 [231] [DEBUG] [app] Ending request: urn:request:77c54656-c65e-40b9-a218-8dd964f9843a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77c54656-c65e-40b9-a218-8dd964f9843a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:22,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:22,086 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:23:22,086 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) notificationworker stdout | 2023-01-17 05:23:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:35.646354+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 05:23:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:35 UTC)" (scheduled at 2023-01-17 05:23:25.646354+00:00) notificationworker stdout | 2023-01-17 05:23:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 25, 647235), True, datetime.datetime(2023, 1, 17, 5, 23, 25, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:28,375 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:23:28,487 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:23:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:23:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:35.853396+00:00 (in 6.889239 seconds) exportactionlogsworker stdout | 2023-01-17 05:23:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:28 UTC)" (scheduled at 2023-01-17 05:23:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:23:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:23:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:23:29,414 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:23:29,704 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:29,716 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:29,716 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:29,727 [60] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2023-01-17 05:23:29,879 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:23:29,903 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:23:30,214 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:23:30,358 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:23:30,943 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:23:30,945 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:23:30,948 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:23:30,950 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:23:30,953 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:23:33,464 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:23:33,767 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:23:34,812 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:23:34,855 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-secscan stdout | 2023-01-17 05:23:34,927 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:23:34,929 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:23:34,933 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:23:35,100 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:23:35,226 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:23:35,234 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:23:35,236 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:23:35,242 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:23:35,244 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:23:35,247 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:23:35,250 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:23:35,255 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:23:35,257 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:23:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:36.635997+00:00 (in 0.989163 seconds) notificationworker stdout | 2023-01-17 05:23:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:45 UTC)" (scheduled at 2023-01-17 05:23:35.646354+00:00) notificationworker stdout | 2023-01-17 05:23:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 35, 647302), True, datetime.datetime(2023, 1, 17, 5, 23, 35, 647302), 0, 'notification/%', 50, 1, 0]) chunkcleanupworker stdout | 2023-01-17 05:23:35,650 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:23:35,667 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:35,667 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:35,667 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:45 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:23:35,825 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} exportactionlogsworker stdout | 2023-01-17 05:23:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:23:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:28.963662+00:00 (in 53.109761 seconds) exportactionlogsworker stdout | 2023-01-17 05:23:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:35 UTC)" (scheduled at 2023-01-17 05:23:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:23:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:23:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 35, 854268), True, datetime.datetime(2023, 1, 17, 5, 23, 35, 854268), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:23:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:23:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:23:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:35 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:23:36,090 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:23:36,153 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:23:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:45.646354+00:00 (in 9.009869 seconds) notificationworker stdout | 2023-01-17 05:23:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:28:36 UTC)" (scheduled at 2023-01-17 05:23:36.635997+00:00) notificationworker stdout | 2023-01-17 05:23:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 23, 36, 636850), 'notification/%']) notificationworker stdout | 2023-01-17 05:23:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 23, 36, 636850), True, datetime.datetime(2023, 1, 17, 5, 23, 36, 636850), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:23:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 23, 36, 636850), True, datetime.datetime(2023, 1, 17, 5, 23, 36, 636850), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 23, 36, 636850), 'notification/%']) notificationworker stdout | 2023-01-17 05:23:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:28:36 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:23:36,686 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:23:37,060 [231] [DEBUG] [app] Starting request: urn:request:2944eda7-61d4-45fb-9f8e-671c90856745 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:37,060 [236] [DEBUG] [app] Starting request: urn:request:1fe563f2-63e7-4d26-879d-fadfa165fb82 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:37,065 [242] [DEBUG] [app] Starting request: urn:request:bfa67833-035c-4271-95a2-6b539b001c7f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:37,065 [230] [DEBUG] [app] Starting request: urn:request:5ac42c16-887e-4304-bdc1-ca742ca045d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:37,065 [242] [DEBUG] [app] Ending request: urn:request:bfa67833-035c-4271-95a2-6b539b001c7f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bfa67833-035c-4271-95a2-6b539b001c7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:23:37,066 [230] [DEBUG] [app] Ending request: urn:request:5ac42c16-887e-4304-bdc1-ca742ca045d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5ac42c16-887e-4304-bdc1-ca742ca045d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:23:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:23:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:37,070 [235] [DEBUG] [app] Starting request: urn:request:0a99efb6-2801-435f-820b-7fbaa36f5964 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:37,070 [231] [DEBUG] [app] Starting request: urn:request:f3d2e2e4-66ef-41fa-80ea-a5a7a72d7ba2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:37,070 [235] [DEBUG] [app] Ending request: urn:request:0a99efb6-2801-435f-820b-7fbaa36f5964 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a99efb6-2801-435f-820b-7fbaa36f5964', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:37,070 [231] [DEBUG] [app] Ending request: urn:request:f3d2e2e4-66ef-41fa-80ea-a5a7a72d7ba2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3d2e2e4-66ef-41fa-80ea-a5a7a72d7ba2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:37,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:37,087 [236] [DEBUG] [app] Ending request: urn:request:1fe563f2-63e7-4d26-879d-fadfa165fb82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1fe563f2-63e7-4d26-879d-fadfa165fb82', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:37,087 [231] [DEBUG] [app] Ending request: urn:request:2944eda7-61d4-45fb-9f8e-671c90856745 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2944eda7-61d4-45fb-9f8e-671c90856745', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:23:37,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) teamsyncworker stdout | 2023-01-17 05:23:37,240 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:23:37,256 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} securityscanningnotificationworker stdout | 2023-01-17 05:23:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:23:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:37.537407+00:00 (in 0.280345 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:23:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:37 UTC)" (scheduled at 2023-01-17 05:23:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:23:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:23:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 37, 257470), True, datetime.datetime(2023, 1, 17, 5, 23, 37, 257470), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:23:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:23:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:23:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:23:37,471 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} securityscanningnotificationworker stdout | 2023-01-17 05:23:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:23:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:37.256652+00:00 (in 59.718882 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:23:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:37 UTC)" (scheduled at 2023-01-17 05:23:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:23:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:23:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:37 UTC)" executed successfully builder stdout | 2023-01-17 05:23:39,730 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:39,742 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:39,742 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:39,752 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:23:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:23:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:10.738568+00:00 (in 29.999546 seconds) buildlogsarchiver stdout | 2023-01-17 05:23:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:10 UTC)" (scheduled at 2023-01-17 05:23:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:23:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 23, 40, 739450), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:23:40,756 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:23:40,756 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:23:40,756 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:23:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:23:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:42.941536+00:00 (in 1.792834 seconds) namespacegcworker stdout | 2023-01-17 05:23:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:41 UTC)" (scheduled at 2023-01-17 05:23:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:23:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:23:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 41, 149135), True, datetime.datetime(2023, 1, 17, 5, 23, 41, 149135), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:23:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:23:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:23:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:23:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:23:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:41.148245+00:00 (in 58.206223 seconds) namespacegcworker stdout | 2023-01-17 05:23:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:42 UTC)" (scheduled at 2023-01-17 05:23:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:23:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:23:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:24:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:55.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:23:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:55 UTC)" (scheduled at 2023-01-17 05:23:45.646354+00:00) notificationworker stdout | 2023-01-17 05:23:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 45, 647214), True, datetime.datetime(2023, 1, 17, 5, 23, 45, 647214), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:23:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:23:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:23:49.537472+00:00 (in 2.600022 seconds) securityworker stdout | 2023-01-17 05:23:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:16 UTC)" (scheduled at 2023-01-17 05:23:46.936988+00:00) securityworker stdout | 2023-01-17 05:23:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:23:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:23:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:16.936988+00:00 (in 27.399027 seconds) securityworker stdout | 2023-01-17 05:23:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:19 UTC)" (scheduled at 2023-01-17 05:23:49.537472+00:00) securityworker stdout | 2023-01-17 05:23:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:23:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:23:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:23:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:23:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:23:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:23:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:23:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:23:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:23:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:23:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:23:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:23:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:23:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:19 UTC)" executed successfully builder stdout | 2023-01-17 05:23:49,762 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:49,780 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:49,780 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:49,791 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:23:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:23:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:21.348605+00:00 (in 29.999530 seconds) gcworker stdout | 2023-01-17 05:23:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:21 UTC)" (scheduled at 2023-01-17 05:23:51.348605+00:00) gcworker stdout | 2023-01-17 05:23:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723431349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:23:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:23:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:23:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:23:52,060 [236] [DEBUG] [app] Starting request: urn:request:1e847737-ea21-430c-87e9-167917a1d683 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:52,060 [231] [DEBUG] [app] Starting request: urn:request:8be24a01-a4dd-41db-ad06-37423e97543e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:23:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:23:52,065 [243] [DEBUG] [app] Starting request: urn:request:040bf074-62b0-405d-ba32-7854a07ffefb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,065 [239] [DEBUG] [app] Starting request: urn:request:e19047a8-2681-41a6-844d-4dcb58166e5d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,065 [239] [DEBUG] [app] Ending request: urn:request:e19047a8-2681-41a6-844d-4dcb58166e5d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e19047a8-2681-41a6-844d-4dcb58166e5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,065 [243] [DEBUG] [app] Ending request: urn:request:040bf074-62b0-405d-ba32-7854a07ffefb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:040bf074-62b0-405d-ba32-7854a07ffefb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:23:52,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:23:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:23:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:23:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:23:52,071 [235] [DEBUG] [app] Starting request: urn:request:ba174e6d-ef3d-4493-a2f3-0ce251223c01 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:52,071 [236] [DEBUG] [app] Starting request: urn:request:c4a66913-1682-4f92-bf27-aa6a327a961b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:23:52,071 [235] [DEBUG] [app] Ending request: urn:request:ba174e6d-ef3d-4493-a2f3-0ce251223c01 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba174e6d-ef3d-4493-a2f3-0ce251223c01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:23:52,071 [236] [DEBUG] [app] Ending request: urn:request:c4a66913-1682-4f92-bf27-aa6a327a961b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4a66913-1682-4f92-bf27-aa6a327a961b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:23:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:23:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:23:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:23:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:23:52,088 [236] [DEBUG] [app] Ending request: urn:request:1e847737-ea21-430c-87e9-167917a1d683 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e847737-ea21-430c-87e9-167917a1d683', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:52,088 [231] [DEBUG] [app] Ending request: urn:request:8be24a01-a4dd-41db-ad06-37423e97543e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8be24a01-a4dd-41db-ad06-37423e97543e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:23:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:23:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:23:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:23:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:23:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:01.636362+00:00 (in 5.989527 seconds) notificationworker stdout | 2023-01-17 05:23:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:05 UTC)" (scheduled at 2023-01-17 05:23:55.646354+00:00) notificationworker stdout | 2023-01-17 05:23:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:23:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 23, 55, 647258), True, datetime.datetime(2023, 1, 17, 5, 23, 55, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:23:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:23:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:23:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:23:58,406 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:23:58,501 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:23:59,445 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:23:59,801 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:23:59,813 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:23:59,813 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:23:59,824 [60] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2023-01-17 05:23:59,894 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:23:59,908 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:24:00,248 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:24:00,387 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:24:00,951 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:24:00,954 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:24:00,956 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:24:00,958 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:24:00,961 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:24:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:05.646354+00:00 (in 4.009475 seconds) notificationworker stdout | 2023-01-17 05:24:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:01 UTC)" (scheduled at 2023-01-17 05:24:01.636362+00:00) notificationworker stdout | 2023-01-17 05:24:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:24:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:24:03,473 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:24:03,799 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:24:04,848 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:24:04,886 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-secscan stdout | 2023-01-17 05:24:04,934 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:24:04,936 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} repositorygcworker stdout | 2023-01-17 05:24:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:24:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:19.346120+00:00 (in 14.408877 seconds) repositorygcworker stdout | 2023-01-17 05:24:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:04 UTC)" (scheduled at 2023-01-17 05:24:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:24:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:24:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 4, 937622), True, datetime.datetime(2023, 1, 17, 5, 24, 4, 937622), 0, 'repositorygc/%', 50, 1, 0]) gunicorn-secscan stdout | 2023-01-17 05:24:04,938 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:24:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:24:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:24:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:04 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:24:05,105 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:24:05,236 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:24:05,240 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:24:05,246 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:24:05,250 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:24:05,253 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:24:05,255 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:24:05,257 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:24:05,263 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:24:05,265 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:24:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:15.646354+00:00 (in 9.999497 seconds) notificationworker stdout | 2023-01-17 05:24:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:15 UTC)" (scheduled at 2023-01-17 05:24:05.646354+00:00) notificationworker stdout | 2023-01-17 05:24:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 5, 647363), True, datetime.datetime(2023, 1, 17, 5, 24, 5, 647363), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:24:05,686 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:24:05,861 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:24:06,098 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:24:06,188 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:24:06,722 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:24:07,059 [236] [DEBUG] [app] Starting request: urn:request:01f06615-73c8-4640-bcfd-32237f3cc8ad (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:07,060 [235] [DEBUG] [app] Starting request: urn:request:3fbae6b2-3833-4140-9c6d-22446767883d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:07,064 [230] [DEBUG] [app] Starting request: urn:request:aff38848-f340-4683-9b1b-e5708c7181b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:07,065 [242] [DEBUG] [app] Starting request: urn:request:5ad2d206-d5ae-4d5b-b753-632087702f2b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:07,065 [230] [DEBUG] [app] Ending request: urn:request:aff38848-f340-4683-9b1b-e5708c7181b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aff38848-f340-4683-9b1b-e5708c7181b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:24:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:24:07,065 [242] [DEBUG] [app] Ending request: urn:request:5ad2d206-d5ae-4d5b-b753-632087702f2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5ad2d206-d5ae-4d5b-b753-632087702f2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:07,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:24:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:24:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:07,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:07,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:07,069 [231] [DEBUG] [app] Starting request: urn:request:9bd339d6-8bf4-443c-9cf4-62163c1fb04a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:07,069 [233] [DEBUG] [app] Starting request: urn:request:4b501ac8-3248-4708-89c5-234722d32432 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:07,070 [231] [DEBUG] [app] Ending request: urn:request:9bd339d6-8bf4-443c-9cf4-62163c1fb04a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9bd339d6-8bf4-443c-9cf4-62163c1fb04a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:07,070 [233] [DEBUG] [app] Ending request: urn:request:4b501ac8-3248-4708-89c5-234722d32432 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b501ac8-3248-4708-89c5-234722d32432', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:24:07,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:24:07,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:07,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:07,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:07,087 [235] [DEBUG] [app] Ending request: urn:request:3fbae6b2-3833-4140-9c6d-22446767883d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3fbae6b2-3833-4140-9c6d-22446767883d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:07,087 [236] [DEBUG] [app] Ending request: urn:request:01f06615-73c8-4640-bcfd-32237f3cc8ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01f06615-73c8-4640-bcfd-32237f3cc8ad', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:24:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) teamsyncworker stdout | 2023-01-17 05:24:07,246 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:24:07,284 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:24:07,506 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:24:09,834 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:09,846 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:09,846 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:09,857 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:24:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:24:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:40.738568+00:00 (in 29.999494 seconds) buildlogsarchiver stdout | 2023-01-17 05:24:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:40 UTC)" (scheduled at 2023-01-17 05:24:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:24:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 24, 10, 739484), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:24:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:24:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:24:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:25.646354+00:00 (in 9.999498 seconds) notificationworker stdout | 2023-01-17 05:24:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:25 UTC)" (scheduled at 2023-01-17 05:24:15.646354+00:00) notificationworker stdout | 2023-01-17 05:24:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 15, 647272), True, datetime.datetime(2023, 1, 17, 5, 24, 15, 647272), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:24:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:19.537472+00:00 (in 2.600010 seconds) securityworker stdout | 2023-01-17 05:24:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:46 UTC)" (scheduled at 2023-01-17 05:24:16.936988+00:00) securityworker stdout | 2023-01-17 05:24:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:24:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:24:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:04.936835+00:00 (in 45.590280 seconds) repositorygcworker stdout | 2023-01-17 05:24:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:19 UTC)" (scheduled at 2023-01-17 05:24:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:24:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:24:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:24:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:46.936988+00:00 (in 27.398933 seconds) securityworker stdout | 2023-01-17 05:24:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:49 UTC)" (scheduled at 2023-01-17 05:24:19.537472+00:00) securityworker stdout | 2023-01-17 05:24:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:24:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:24:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:24:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:24:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:24:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:24:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:49 UTC)" executed successfully builder stdout | 2023-01-17 05:24:19,857 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:19,869 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:19,870 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:19,880 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:24:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:24:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:51.348605+00:00 (in 29.999515 seconds) gcworker stdout | 2023-01-17 05:24:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:51 UTC)" (scheduled at 2023-01-17 05:24:21.348605+00:00) gcworker stdout | 2023-01-17 05:24:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723461349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:24:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:24:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:24:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:24:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:24:22,059 [236] [DEBUG] [app] Starting request: urn:request:db9cfa8d-e808-4bf1-828e-77609997e6aa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:22,059 [231] [DEBUG] [app] Starting request: urn:request:452ed2c3-bda4-4152-b15e-8c8fa31be1a8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:22,065 [243] [DEBUG] [app] Starting request: urn:request:ba0b540c-091a-4e07-921c-16561f5bd93a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:22,065 [242] [DEBUG] [app] Starting request: urn:request:023a040c-f841-4c9c-9a7e-065cfca7c3ca (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:22,065 [243] [DEBUG] [app] Ending request: urn:request:ba0b540c-091a-4e07-921c-16561f5bd93a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ba0b540c-091a-4e07-921c-16561f5bd93a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:24:22,065 [242] [DEBUG] [app] Ending request: urn:request:023a040c-f841-4c9c-9a7e-065cfca7c3ca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:023a040c-f841-4c9c-9a7e-065cfca7c3ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:24:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:24:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:22,070 [233] [DEBUG] [app] Starting request: urn:request:6fde8f0e-d029-4dee-9779-e4b113f3b53a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:22,070 [235] [DEBUG] [app] Starting request: urn:request:3a867911-3ac4-46aa-9939-86d52aca4653 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:22,070 [233] [DEBUG] [app] Ending request: urn:request:6fde8f0e-d029-4dee-9779-e4b113f3b53a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6fde8f0e-d029-4dee-9779-e4b113f3b53a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:22,070 [235] [DEBUG] [app] Ending request: urn:request:3a867911-3ac4-46aa-9939-86d52aca4653 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3a867911-3ac4-46aa-9939-86d52aca4653', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:22,084 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:22,087 [231] [DEBUG] [app] Ending request: urn:request:452ed2c3-bda4-4152-b15e-8c8fa31be1a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:452ed2c3-bda4-4152-b15e-8c8fa31be1a8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:24:22,093 [236] [DEBUG] [app] Ending request: urn:request:db9cfa8d-e808-4bf1-828e-77609997e6aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db9cfa8d-e808-4bf1-828e-77609997e6aa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:22,093 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:22,093 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) notificationworker stdout | 2023-01-17 05:24:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:35.646354+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 05:24:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:35 UTC)" (scheduled at 2023-01-17 05:24:25.646354+00:00) notificationworker stdout | 2023-01-17 05:24:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 25, 647190), True, datetime.datetime(2023, 1, 17, 5, 24, 25, 647190), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:28,420 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:24:28,530 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:24:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:24:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:35.853396+00:00 (in 6.889291 seconds) exportactionlogsworker stdout | 2023-01-17 05:24:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:28 UTC)" (scheduled at 2023-01-17 05:24:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:24:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:24:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:24:29,479 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:24:29,882 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:29,894 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:29,894 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:29,905 [60] [DEBUG] [data.database] Disconnecting from database. logrotateworker stdout | 2023-01-17 05:24:29,912 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:24:29,914 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:24:30,283 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:24:30,410 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:24:30,957 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:24:30,960 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:24:30,962 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:24:30,965 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:24:30,967 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:24:33,501 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:24:33,835 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:24:34,884 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:24:34,892 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-secscan stdout | 2023-01-17 05:24:34,941 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:24:34,943 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:24:34,946 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:24:35,111 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:24:35,247 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:24:35,250 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:24:35,255 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:24:35,258 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:24:35,260 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:24:35,263 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:24:35,265 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:24:35,270 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:24:35,274 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:24:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:45.646354+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 05:24:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:45 UTC)" (scheduled at 2023-01-17 05:24:35.646354+00:00) notificationworker stdout | 2023-01-17 05:24:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 35, 647252), True, datetime.datetime(2023, 1, 17, 5, 24, 35, 647252), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:45 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:24:35,721 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} exportactionlogsworker stdout | 2023-01-17 05:24:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:24:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:28.963662+00:00 (in 53.109780 seconds) exportactionlogsworker stdout | 2023-01-17 05:24:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:35 UTC)" (scheduled at 2023-01-17 05:24:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:24:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:24:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 35, 854253), True, datetime.datetime(2023, 1, 17, 5, 24, 35, 854253), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:24:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:24:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:24:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:35 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:24:35,870 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:24:36,113 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:24:36,223 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:24:36,757 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:24:37,059 [231] [DEBUG] [app] Starting request: urn:request:e708f249-5dee-4708-8ddd-cb90a8a3a6fa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:37,060 [233] [DEBUG] [app] Starting request: urn:request:16b159db-123b-4c8d-9f6c-350160b962b6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:37,064 [242] [DEBUG] [app] Starting request: urn:request:093c0d4c-8da5-4d7f-b82b-5d5df261672c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:37,065 [241] [DEBUG] [app] Starting request: urn:request:9f019402-59bd-47ac-9135-2d7ce4e08d11 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:37,065 [242] [DEBUG] [app] Ending request: urn:request:093c0d4c-8da5-4d7f-b82b-5d5df261672c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:093c0d4c-8da5-4d7f-b82b-5d5df261672c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:24:37,065 [241] [DEBUG] [app] Ending request: urn:request:9f019402-59bd-47ac-9135-2d7ce4e08d11 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9f019402-59bd-47ac-9135-2d7ce4e08d11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:24:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:24:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:24:37,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:37,070 [235] [DEBUG] [app] Starting request: urn:request:76a907bb-3880-46ea-a343-7a8464133854 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:37,070 [231] [DEBUG] [app] Starting request: urn:request:afb3877a-a1f8-474a-8583-8fbc9a6e5cf9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:37,070 [235] [DEBUG] [app] Ending request: urn:request:76a907bb-3880-46ea-a343-7a8464133854 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76a907bb-3880-46ea-a343-7a8464133854', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:37,071 [231] [DEBUG] [app] Ending request: urn:request:afb3877a-a1f8-474a-8583-8fbc9a6e5cf9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:afb3877a-a1f8-474a-8583-8fbc9a6e5cf9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:37,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:37,088 [231] [DEBUG] [app] Ending request: urn:request:e708f249-5dee-4708-8ddd-cb90a8a3a6fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e708f249-5dee-4708-8ddd-cb90a8a3a6fa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:37,088 [233] [DEBUG] [app] Ending request: urn:request:16b159db-123b-4c8d-9f6c-350160b962b6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16b159db-123b-4c8d-9f6c-350160b962b6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:37,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:24:37,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:24:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:24:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:37.537407+00:00 (in 0.280368 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:24:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:37 UTC)" (scheduled at 2023-01-17 05:24:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:24:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:24:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 37, 257475), True, datetime.datetime(2023, 1, 17, 5, 24, 37, 257475), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:24:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:24:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:24:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:24:37,282 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:24:37,319 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} securityscanningnotificationworker stdout | 2023-01-17 05:24:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:24:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:37.256652+00:00 (in 59.718824 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:24:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:37 UTC)" (scheduled at 2023-01-17 05:24:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:24:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:24:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:24:37,541 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:24:39,915 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:39,927 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:39,927 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:39,938 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:24:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:24:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:10.738568+00:00 (in 29.999492 seconds) buildlogsarchiver stdout | 2023-01-17 05:24:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:10 UTC)" (scheduled at 2023-01-17 05:24:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:24:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 24, 40, 739497), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:24:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:24:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:24:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:24:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:24:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:42.941536+00:00 (in 1.792828 seconds) namespacegcworker stdout | 2023-01-17 05:24:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:41 UTC)" (scheduled at 2023-01-17 05:24:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:24:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:24:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 41, 149117), True, datetime.datetime(2023, 1, 17, 5, 24, 41, 149117), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:24:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:24:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:24:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:24:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:24:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:41.148245+00:00 (in 58.206254 seconds) namespacegcworker stdout | 2023-01-17 05:24:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:42 UTC)" (scheduled at 2023-01-17 05:24:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:24:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:24:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:25:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:55.646354+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:24:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:55 UTC)" (scheduled at 2023-01-17 05:24:45.646354+00:00) notificationworker stdout | 2023-01-17 05:24:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 45, 647196), True, datetime.datetime(2023, 1, 17, 5, 24, 45, 647196), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:24:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:24:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:24:49.537472+00:00 (in 2.600049 seconds) securityworker stdout | 2023-01-17 05:24:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:16 UTC)" (scheduled at 2023-01-17 05:24:46.936988+00:00) securityworker stdout | 2023-01-17 05:24:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:24:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:24:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:16.936988+00:00 (in 27.399001 seconds) securityworker stdout | 2023-01-17 05:24:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:19 UTC)" (scheduled at 2023-01-17 05:24:49.537472+00:00) securityworker stdout | 2023-01-17 05:24:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:24:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:24:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:24:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:24:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:24:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:24:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:24:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:24:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:24:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:24:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:24:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:24:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:24:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:19 UTC)" executed successfully builder stdout | 2023-01-17 05:24:49,945 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:49,957 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:49,957 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:24:49,967 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:24:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:24:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:21.348605+00:00 (in 29.999528 seconds) gcworker stdout | 2023-01-17 05:24:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:21 UTC)" (scheduled at 2023-01-17 05:24:51.348605+00:00) gcworker stdout | 2023-01-17 05:24:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723491349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:24:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:24:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:24:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:24:52,060 [236] [DEBUG] [app] Starting request: urn:request:62ec92f4-8020-4632-8671-e860d897178e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:52,060 [231] [DEBUG] [app] Starting request: urn:request:a2bc0d91-f5ae-4c1b-9ed4-aec989ab4ae4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:24:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:24:52,065 [243] [DEBUG] [app] Starting request: urn:request:873f10be-e070-44f4-a0fd-de61014b932b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:52,065 [239] [DEBUG] [app] Starting request: urn:request:81bc6555-9245-40cb-9584-c03ec2f62b6d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:24:52,065 [243] [DEBUG] [app] Ending request: urn:request:873f10be-e070-44f4-a0fd-de61014b932b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:873f10be-e070-44f4-a0fd-de61014b932b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:24:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:24:52,065 [239] [DEBUG] [app] Ending request: urn:request:81bc6555-9245-40cb-9584-c03ec2f62b6d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81bc6555-9245-40cb-9584-c03ec2f62b6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:24:52,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:24:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:24:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:24:52,069 [235] [DEBUG] [app] Starting request: urn:request:051fcb0f-17f9-4843-8da1-92c6811b76ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:52,070 [235] [DEBUG] [app] Ending request: urn:request:051fcb0f-17f9-4843-8da1-92c6811b76ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:051fcb0f-17f9-4843-8da1-92c6811b76ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:24:52,070 [233] [DEBUG] [app] Starting request: urn:request:ed324423-4085-4d0f-8ca3-f234d0daf252 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:24:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:52,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:24:52,070 [233] [DEBUG] [app] Ending request: urn:request:ed324423-4085-4d0f-8ca3-f234d0daf252 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ed324423-4085-4d0f-8ca3-f234d0daf252', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:24:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:24:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:24:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:24:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:24:52,087 [236] [DEBUG] [app] Ending request: urn:request:62ec92f4-8020-4632-8671-e860d897178e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62ec92f4-8020-4632-8671-e860d897178e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:52,087 [231] [DEBUG] [app] Ending request: urn:request:a2bc0d91-f5ae-4c1b-9ed4-aec989ab4ae4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2bc0d91-f5ae-4c1b-9ed4-aec989ab4ae4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:24:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:24:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:24:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:24:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) notificationworker stdout | 2023-01-17 05:24:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:24:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:01.636362+00:00 (in 5.989566 seconds) notificationworker stdout | 2023-01-17 05:24:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:05 UTC)" (scheduled at 2023-01-17 05:24:55.646354+00:00) notificationworker stdout | 2023-01-17 05:24:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:24:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 24, 55, 647163), True, datetime.datetime(2023, 1, 17, 5, 24, 55, 647163), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:24:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:24:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:24:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:24:58,455 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:24:58,566 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:24:59,496 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:24:59,920 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:24:59,943 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 05:24:59,977 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:24:59,989 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:24:59,989 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:00,000 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:25:00,289 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:25:00,440 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:25:00,965 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:25:00,967 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:25:00,969 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:25:00,972 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:25:00,975 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:25:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:05.646354+00:00 (in 4.009491 seconds) notificationworker stdout | 2023-01-17 05:25:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:01 UTC)" (scheduled at 2023-01-17 05:25:01.636362+00:00) notificationworker stdout | 2023-01-17 05:25:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:25:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:25:03,536 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:25:03,864 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:25:04,914 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:25:04,927 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:25:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:25:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:19.346120+00:00 (in 14.408863 seconds) repositorygcworker stdout | 2023-01-17 05:25:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:04 UTC)" (scheduled at 2023-01-17 05:25:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:25:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:25:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 4, 937671), True, datetime.datetime(2023, 1, 17, 5, 25, 4, 937671), 0, 'repositorygc/%', 50, 1, 0]) gunicorn-secscan stdout | 2023-01-17 05:25:04,948 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} repositorygcworker stdout | 2023-01-17 05:25:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:25:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:25:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:25:04,950 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:25:04,953 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} repositorygcworker stdout | 2023-01-17 05:25:05,144 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:25:05,256 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:25:05,258 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:25:05,263 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:25:05,265 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:25:05,268 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:25:05,271 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:25:05,273 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:25:05,277 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:25:05,282 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:25:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:15.646354+00:00 (in 9.999294 seconds) notificationworker stdout | 2023-01-17 05:25:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:15 UTC)" (scheduled at 2023-01-17 05:25:05.646354+00:00) notificationworker stdout | 2023-01-17 05:25:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 5, 647493), True, datetime.datetime(2023, 1, 17, 5, 25, 5, 647493), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:25:05,756 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:25:05,902 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:25:06,129 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:25:06,258 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:25:06,788 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:25:07,059 [231] [DEBUG] [app] Starting request: urn:request:d5ec1369-ca4c-4524-add7-b054bf68267f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:07,060 [236] [DEBUG] [app] Starting request: urn:request:8f45a729-c20e-47ef-a067-d4a95b72bfe1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:07,064 [243] [DEBUG] [app] Starting request: urn:request:4550b6ec-ceba-4de8-939d-8c99923b05fe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:07,065 [242] [DEBUG] [app] Starting request: urn:request:51204117-f5ed-45a8-8086-dfce763e50c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:07,065 [243] [DEBUG] [app] Ending request: urn:request:4550b6ec-ceba-4de8-939d-8c99923b05fe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4550b6ec-ceba-4de8-939d-8c99923b05fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:25:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:25:07,065 [242] [DEBUG] [app] Ending request: urn:request:51204117-f5ed-45a8-8086-dfce763e50c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51204117-f5ed-45a8-8086-dfce763e50c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:25:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:07,070 [235] [DEBUG] [app] Starting request: urn:request:349065ae-9a7e-4ed8-9290-b8febaca4e1a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:07,070 [236] [DEBUG] [app] Starting request: urn:request:27e22850-cf75-4de3-8348-7626a62cd5b4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:07,071 [235] [DEBUG] [app] Ending request: urn:request:349065ae-9a7e-4ed8-9290-b8febaca4e1a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:349065ae-9a7e-4ed8-9290-b8febaca4e1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:07,071 [236] [DEBUG] [app] Ending request: urn:request:27e22850-cf75-4de3-8348-7626a62cd5b4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27e22850-cf75-4de3-8348-7626a62cd5b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:25:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:07,088 [231] [DEBUG] [app] Ending request: urn:request:d5ec1369-ca4c-4524-add7-b054bf68267f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d5ec1369-ca4c-4524-add7-b054bf68267f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:07,088 [236] [DEBUG] [app] Ending request: urn:request:8f45a729-c20e-47ef-a067-d4a95b72bfe1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f45a729-c20e-47ef-a067-d4a95b72bfe1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:25:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) teamsyncworker stdout | 2023-01-17 05:25:07,318 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:25:07,354 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:25:07,576 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:25:10,001 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:10,012 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:10,013 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:10,023 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:25:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:25:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:40.738568+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2023-01-17 05:25:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:40 UTC)" (scheduled at 2023-01-17 05:25:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:25:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 25, 10, 739392), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:25:10,756 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:25:10,756 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:25:10,756 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:25.646354+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 05:25:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:25 UTC)" (scheduled at 2023-01-17 05:25:15.646354+00:00) notificationworker stdout | 2023-01-17 05:25:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 15, 647205), True, datetime.datetime(2023, 1, 17, 5, 25, 15, 647205), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:25:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:19.537472+00:00 (in 2.599999 seconds) securityworker stdout | 2023-01-17 05:25:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:46 UTC)" (scheduled at 2023-01-17 05:25:16.936988+00:00) securityworker stdout | 2023-01-17 05:25:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:25:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:25:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:04.936835+00:00 (in 45.590266 seconds) repositorygcworker stdout | 2023-01-17 05:25:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:19 UTC)" (scheduled at 2023-01-17 05:25:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:25:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:25:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:25:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:46.936988+00:00 (in 27.399013 seconds) securityworker stdout | 2023-01-17 05:25:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:49 UTC)" (scheduled at 2023-01-17 05:25:19.537472+00:00) securityworker stdout | 2023-01-17 05:25:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:25:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:25:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:25:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:25:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:25:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:25:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:49 UTC)" executed successfully builder stdout | 2023-01-17 05:25:20,033 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:20,046 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:20,046 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:20,056 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:25:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:25:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:51.348605+00:00 (in 29.999521 seconds) gcworker stdout | 2023-01-17 05:25:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:51 UTC)" (scheduled at 2023-01-17 05:25:21.348605+00:00) gcworker stdout | 2023-01-17 05:25:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723521349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:25:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:25:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:25:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:25:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:25:22,059 [235] [DEBUG] [app] Starting request: urn:request:0bfd105a-e05b-4982-850d-fcfdcecfae32 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:22,059 [236] [DEBUG] [app] Starting request: urn:request:8b2ae44b-e9b6-4a16-aea8-4ff67d64c77f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:22,060 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:22,064 [242] [DEBUG] [app] Starting request: urn:request:2e3c2c0f-e2cf-4b64-a758-659765c299b7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:22,064 [242] [DEBUG] [app] Ending request: urn:request:2e3c2c0f-e2cf-4b64-a758-659765c299b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2e3c2c0f-e2cf-4b64-a758-659765c299b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:22,064 [230] [DEBUG] [app] Starting request: urn:request:09339f13-fac3-45e6-bf31-31474e5762e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:25:22,065 [230] [DEBUG] [app] Ending request: urn:request:09339f13-fac3-45e6-bf31-31474e5762e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09339f13-fac3-45e6-bf31-31474e5762e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:22,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:25:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:25:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:22,070 [233] [DEBUG] [app] Starting request: urn:request:bf4479af-a03f-4046-96ab-eaaa29728e19 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:22,070 [236] [DEBUG] [app] Starting request: urn:request:3b2b9cd8-d14c-4d46-a972-d721d9828c1f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:22,070 [233] [DEBUG] [app] Ending request: urn:request:bf4479af-a03f-4046-96ab-eaaa29728e19 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf4479af-a03f-4046-96ab-eaaa29728e19', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:22,071 [236] [DEBUG] [app] Ending request: urn:request:3b2b9cd8-d14c-4d46-a972-d721d9828c1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b2b9cd8-d14c-4d46-a972-d721d9828c1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:22,088 [236] [DEBUG] [app] Ending request: urn:request:8b2ae44b-e9b6-4a16-aea8-4ff67d64c77f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b2ae44b-e9b6-4a16-aea8-4ff67d64c77f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:22,088 [235] [DEBUG] [app] Ending request: urn:request:0bfd105a-e05b-4982-850d-fcfdcecfae32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0bfd105a-e05b-4982-850d-fcfdcecfae32', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:25:22,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:25:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:25:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:35.646354+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:25:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:35 UTC)" (scheduled at 2023-01-17 05:25:25.646354+00:00) notificationworker stdout | 2023-01-17 05:25:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 25, 647218), True, datetime.datetime(2023, 1, 17, 5, 25, 25, 647218), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:28,476 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:25:28,602 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:25:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:25:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:35.853396+00:00 (in 6.889312 seconds) exportactionlogsworker stdout | 2023-01-17 05:25:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:28 UTC)" (scheduled at 2023-01-17 05:25:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:25:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:25:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:25:29,532 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:25:29,949 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:25:29,963 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 05:25:30,063 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:30,076 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:30,076 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:30,087 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:25:30,325 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:25:30,476 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:25:30,972 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:25:30,975 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:25:30,978 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:25:30,980 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:25:30,983 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:25:33,571 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:25:33,871 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} servicekey stdout | 2023-01-17 05:25:34,950 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:25:34,955 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:25:34,958 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:25:34,961 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:25:34,963 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:25:35,172 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:25:35,265 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:25:35,267 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:25:35,272 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:25:35,275 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:25:35,277 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:25:35,280 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:25:35,282 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:25:35,284 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:25:35,290 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:25:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:45.646354+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 05:25:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:45 UTC)" (scheduled at 2023-01-17 05:25:35.646354+00:00) notificationworker stdout | 2023-01-17 05:25:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 35, 647274), True, datetime.datetime(2023, 1, 17, 5, 25, 35, 647274), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:45 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:25:35,776 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} exportactionlogsworker stdout | 2023-01-17 05:25:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:25:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:28.963662+00:00 (in 53.109817 seconds) exportactionlogsworker stdout | 2023-01-17 05:25:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:35 UTC)" (scheduled at 2023-01-17 05:25:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:25:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:25:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 35, 854219), True, datetime.datetime(2023, 1, 17, 5, 25, 35, 854219), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:25:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:25:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:25:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:35 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:25:35,909 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:25:36,151 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:25:36,279 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:25:36,824 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:25:37,060 [235] [DEBUG] [app] Starting request: urn:request:efa3676d-e807-48bb-b0a4-9bbd43169aff (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:37,060 [236] [DEBUG] [app] Starting request: urn:request:3d5a8a8b-8d20-4809-860b-ccfa356c1766 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:37,065 [241] [DEBUG] [app] Starting request: urn:request:d13d8eaf-cc73-4d5b-a353-2accd83f1821 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:37,065 [243] [DEBUG] [app] Starting request: urn:request:2ede8aef-8164-4c23-b9bc-dd079b2ecdcf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:37,065 [241] [DEBUG] [app] Ending request: urn:request:d13d8eaf-cc73-4d5b-a353-2accd83f1821 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d13d8eaf-cc73-4d5b-a353-2accd83f1821', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:25:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:25:37,066 [243] [DEBUG] [app] Ending request: urn:request:2ede8aef-8164-4c23-b9bc-dd079b2ecdcf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ede8aef-8164-4c23-b9bc-dd079b2ecdcf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:25:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:25:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:37,071 [231] [DEBUG] [app] Starting request: urn:request:0b0e523c-99aa-42c5-bb92-9d799858473b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:37,071 [236] [DEBUG] [app] Starting request: urn:request:936b7d83-b2fd-410b-bd69-48c20bdb8b9e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:37,071 [231] [DEBUG] [app] Ending request: urn:request:0b0e523c-99aa-42c5-bb92-9d799858473b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b0e523c-99aa-42c5-bb92-9d799858473b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:37,071 [236] [DEBUG] [app] Ending request: urn:request:936b7d83-b2fd-410b-bd69-48c20bdb8b9e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:936b7d83-b2fd-410b-bd69-48c20bdb8b9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:37,088 [235] [DEBUG] [app] Ending request: urn:request:efa3676d-e807-48bb-b0a4-9bbd43169aff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:efa3676d-e807-48bb-b0a4-9bbd43169aff', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:37,088 [236] [DEBUG] [app] Ending request: urn:request:3d5a8a8b-8d20-4809-860b-ccfa356c1766 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d5a8a8b-8d20-4809-860b-ccfa356c1766', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:25:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:25:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:25:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:37.537407+00:00 (in 0.280327 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:25:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:37 UTC)" (scheduled at 2023-01-17 05:25:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:25:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:25:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 37, 257521), True, datetime.datetime(2023, 1, 17, 5, 25, 37, 257521), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:25:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:25:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:25:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:25:37,354 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:25:37,389 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} securityscanningnotificationworker stdout | 2023-01-17 05:25:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:25:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:23.537048+00:00 (in 45.999184 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:25:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:37 UTC)" (scheduled at 2023-01-17 05:25:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:25:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:25:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:25:37,612 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:25:40,092 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:40,104 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:40,105 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:40,115 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:25:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:25:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:10.738568+00:00 (in 29.999575 seconds) buildlogsarchiver stdout | 2023-01-17 05:25:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:10 UTC)" (scheduled at 2023-01-17 05:25:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:25:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 25, 40, 739376), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:25:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:25:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:25:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:25:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:25:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:42.941536+00:00 (in 1.792809 seconds) namespacegcworker stdout | 2023-01-17 05:25:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:41 UTC)" (scheduled at 2023-01-17 05:25:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:25:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:25:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 41, 149141), True, datetime.datetime(2023, 1, 17, 5, 25, 41, 149141), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:25:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:25:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:25:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:25:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:25:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:41.148245+00:00 (in 58.206229 seconds) namespacegcworker stdout | 2023-01-17 05:25:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:42 UTC)" (scheduled at 2023-01-17 05:25:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:25:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:25:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:26:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:55.646354+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 05:25:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:55 UTC)" (scheduled at 2023-01-17 05:25:45.646354+00:00) notificationworker stdout | 2023-01-17 05:25:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 45, 647236), True, datetime.datetime(2023, 1, 17, 5, 25, 45, 647236), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:25:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:25:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:25:49.537472+00:00 (in 2.599988 seconds) securityworker stdout | 2023-01-17 05:25:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:16 UTC)" (scheduled at 2023-01-17 05:25:46.936988+00:00) securityworker stdout | 2023-01-17 05:25:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:25:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:25:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:16.936988+00:00 (in 27.399005 seconds) securityworker stdout | 2023-01-17 05:25:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:19 UTC)" (scheduled at 2023-01-17 05:25:49.537472+00:00) securityworker stdout | 2023-01-17 05:25:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:25:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:25:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:25:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:25:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:25:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:25:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:25:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:25:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:25:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:25:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:25:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:25:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:25:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:19 UTC)" executed successfully builder stdout | 2023-01-17 05:25:50,125 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:25:50,137 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:25:50,137 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:25:50,148 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:25:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:25:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:21.348605+00:00 (in 29.999525 seconds) gcworker stdout | 2023-01-17 05:25:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:21 UTC)" (scheduled at 2023-01-17 05:25:51.348605+00:00) gcworker stdout | 2023-01-17 05:25:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723551349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:25:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:25:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:25:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:25:52,060 [231] [DEBUG] [app] Starting request: urn:request:f923dbeb-b082-4c74-93a6-e9db5ba16cc6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:52,061 [235] [DEBUG] [app] Starting request: urn:request:3266d4a0-14b0-40f1-ae8a-c68c53d57769 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:25:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:52,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:25:52,066 [243] [DEBUG] [app] Starting request: urn:request:5babcc72-98fa-49e5-9778-f56fe91d81e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,066 [242] [DEBUG] [app] Starting request: urn:request:a2da47d3-46a2-4618-a904-592bafd78102 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,066 [243] [DEBUG] [app] Ending request: urn:request:5babcc72-98fa-49e5-9778-f56fe91d81e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5babcc72-98fa-49e5-9778-f56fe91d81e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,066 [242] [DEBUG] [app] Ending request: urn:request:a2da47d3-46a2-4618-a904-592bafd78102 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2da47d3-46a2-4618-a904-592bafd78102', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:25:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:25:52,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:52,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:52,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:25:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:25:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:25:52,071 [236] [DEBUG] [app] Starting request: urn:request:1d38f3c1-e461-43fe-afb8-f61694d9ac0d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:52,071 [231] [DEBUG] [app] Starting request: urn:request:8b626016-396c-464f-ba7c-3694201e361a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:25:52,072 [236] [DEBUG] [app] Ending request: urn:request:1d38f3c1-e461-43fe-afb8-f61694d9ac0d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d38f3c1-e461-43fe-afb8-f61694d9ac0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:25:52,072 [231] [DEBUG] [app] Ending request: urn:request:8b626016-396c-464f-ba7c-3694201e361a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b626016-396c-464f-ba7c-3694201e361a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:25:52,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:25:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:25:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:25:52,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:25:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:52,082 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:25:52,088 [231] [DEBUG] [app] Ending request: urn:request:f923dbeb-b082-4c74-93a6-e9db5ba16cc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f923dbeb-b082-4c74-93a6-e9db5ba16cc6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:25:52,091 [235] [DEBUG] [app] Ending request: urn:request:3266d4a0-14b0-40f1-ae8a-c68c53d57769 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3266d4a0-14b0-40f1-ae8a-c68c53d57769', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:25:52,091 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:25:52,092 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:25:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.032) notificationworker stdout | 2023-01-17 05:25:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:25:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:01.636362+00:00 (in 5.989541 seconds) notificationworker stdout | 2023-01-17 05:25:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:05 UTC)" (scheduled at 2023-01-17 05:25:55.646354+00:00) notificationworker stdout | 2023-01-17 05:25:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:25:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 25, 55, 647256), True, datetime.datetime(2023, 1, 17, 5, 25, 55, 647256), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:25:55,660 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:25:55,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:25:55,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:25:58,504 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:25:58,615 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:25:59,539 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:25:59,979 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:25:59,997 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 05:26:00,158 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:00,170 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:00,171 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:00,181 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:26:00,360 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:26:00,487 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:26:00,980 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:26:00,983 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:26:00,986 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:26:00,988 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:26:00,990 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:26:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:05.646354+00:00 (in 4.009523 seconds) notificationworker stdout | 2023-01-17 05:26:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:01 UTC)" (scheduled at 2023-01-17 05:26:01.636362+00:00) notificationworker stdout | 2023-01-17 05:26:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:26:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:26:03,608 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:26:03,880 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:26:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:26:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:17.345601+00:00 (in 12.408298 seconds) repositorygcworker stdout | 2023-01-17 05:26:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:04 UTC)" (scheduled at 2023-01-17 05:26:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:26:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:26:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 4, 937803), True, datetime.datetime(2023, 1, 17, 5, 26, 4, 937803), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:26:04,951 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:26:04,951 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:26:04,951 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:04 UTC)" executed successfully servicekey stdout | 2023-01-17 05:26:04,957 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-secscan stdout | 2023-01-17 05:26:04,963 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:26:04,965 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:26:04,968 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:26:04,990 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:26:05,192 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:26:05,273 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:26:05,276 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:26:05,279 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:26:05,282 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:26:05,285 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:26:05,287 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:26:05,290 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:26:05,292 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:26:05,299 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:26:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:15.646354+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 05:26:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:15 UTC)" (scheduled at 2023-01-17 05:26:05.646354+00:00) notificationworker stdout | 2023-01-17 05:26:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 5, 647255), True, datetime.datetime(2023, 1, 17, 5, 26, 5, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:26:05,811 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:26:05,944 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:26:06,185 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:26:06,311 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:26:06,859 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:26:07,060 [231] [DEBUG] [app] Starting request: urn:request:8abae816-8643-47eb-9b1f-8bc9a6b5ae59 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:07,060 [236] [DEBUG] [app] Starting request: urn:request:1402daa8-de31-40a9-bdec-b7aaf87bc5ae (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:07,065 [242] [DEBUG] [app] Starting request: urn:request:1fb52460-2dbe-449c-915c-2c6c30d4f8ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:07,065 [230] [DEBUG] [app] Starting request: urn:request:d6950709-c64b-4a42-9080-d7e7435c52b4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:07,065 [242] [DEBUG] [app] Ending request: urn:request:1fb52460-2dbe-449c-915c-2c6c30d4f8ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1fb52460-2dbe-449c-915c-2c6c30d4f8ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:07,066 [230] [DEBUG] [app] Ending request: urn:request:d6950709-c64b-4a42-9080-d7e7435c52b4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6950709-c64b-4a42-9080-d7e7435c52b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:26:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:26:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:07,070 [233] [DEBUG] [app] Starting request: urn:request:ce7f6520-23a1-4ac5-8b7d-f04b081a8eda (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:07,070 [233] [DEBUG] [app] Ending request: urn:request:ce7f6520-23a1-4ac5-8b7d-f04b081a8eda (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ce7f6520-23a1-4ac5-8b7d-f04b081a8eda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:07,070 [235] [DEBUG] [app] Starting request: urn:request:010374f9-9bf9-4bab-b0d4-3379391a4fa1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:07,071 [235] [DEBUG] [app] Ending request: urn:request:010374f9-9bf9-4bab-b0d4-3379391a4fa1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:010374f9-9bf9-4bab-b0d4-3379391a4fa1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:26:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:07,087 [236] [DEBUG] [app] Ending request: urn:request:1402daa8-de31-40a9-bdec-b7aaf87bc5ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1402daa8-de31-40a9-bdec-b7aaf87bc5ae', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:07,087 [231] [DEBUG] [app] Ending request: urn:request:8abae816-8643-47eb-9b1f-8bc9a6b5ae59 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8abae816-8643-47eb-9b1f-8bc9a6b5ae59', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:26:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) teamsyncworker stdout | 2023-01-17 05:26:07,390 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:26:07,415 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:26:07,647 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:26:10,183 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:10,195 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:10,195 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:10,206 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:26:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:26:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:40.738568+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 05:26:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:40 UTC)" (scheduled at 2023-01-17 05:26:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:26:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 26, 10, 739469), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:26:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:26:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:26:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:25.646354+00:00 (in 9.999511 seconds) notificationworker stdout | 2023-01-17 05:26:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:25 UTC)" (scheduled at 2023-01-17 05:26:15.646354+00:00) notificationworker stdout | 2023-01-17 05:26:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 15, 647241), True, datetime.datetime(2023, 1, 17, 5, 26, 15, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:26:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:19.537472+00:00 (in 2.600015 seconds) securityworker stdout | 2023-01-17 05:26:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:46 UTC)" (scheduled at 2023-01-17 05:26:16.936988+00:00) securityworker stdout | 2023-01-17 05:26:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:26:17,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:26:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:19.346120+00:00 (in 1.999979 seconds) repositorygcworker stdout | 2023-01-17 05:26:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:17 UTC)" (scheduled at 2023-01-17 05:26:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:26:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 26, 17, 346554), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:26:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 17, 346554), True, datetime.datetime(2023, 1, 17, 5, 26, 17, 346554), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:26:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 17, 346554), True, datetime.datetime(2023, 1, 17, 5, 26, 17, 346554), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 26, 17, 346554), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:26:17,365 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:26:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:26:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:26:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:04.936835+00:00 (in 45.590233 seconds) repositorygcworker stdout | 2023-01-17 05:26:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:19 UTC)" (scheduled at 2023-01-17 05:26:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:26:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:26:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:26:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:46.936988+00:00 (in 27.399043 seconds) securityworker stdout | 2023-01-17 05:26:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:49 UTC)" (scheduled at 2023-01-17 05:26:19.537472+00:00) securityworker stdout | 2023-01-17 05:26:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:26:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:26:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:26:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:26:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:26:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:26:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:49 UTC)" executed successfully builder stdout | 2023-01-17 05:26:20,216 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:20,228 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:20,228 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:20,239 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:26:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:26:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:51.348605+00:00 (in 29.999519 seconds) gcworker stdout | 2023-01-17 05:26:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:51 UTC)" (scheduled at 2023-01-17 05:26:21.348605+00:00) gcworker stdout | 2023-01-17 05:26:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723581349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:26:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:26:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:26:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:26:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:26:22,060 [231] [DEBUG] [app] Starting request: urn:request:fa3330eb-b133-47aa-83c9-0de4e0e0009c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:22,060 [236] [DEBUG] [app] Starting request: urn:request:c309e458-c0b4-4e8c-9400-d593492c4879 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:22,065 [241] [DEBUG] [app] Starting request: urn:request:9790cea5-002e-4dbf-9826-02d6c09b9759 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:22,065 [230] [DEBUG] [app] Starting request: urn:request:0f3ae8b4-d121-49bd-b8e8-84170ede62b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:22,065 [241] [DEBUG] [app] Ending request: urn:request:9790cea5-002e-4dbf-9826-02d6c09b9759 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9790cea5-002e-4dbf-9826-02d6c09b9759', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:26:22,066 [230] [DEBUG] [app] Ending request: urn:request:0f3ae8b4-d121-49bd-b8e8-84170ede62b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f3ae8b4-d121-49bd-b8e8-84170ede62b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:26:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:26:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:22,070 [233] [DEBUG] [app] Starting request: urn:request:64795ca4-cfaa-47ad-a48e-f7fc36f33a50 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:22,070 [236] [DEBUG] [app] Starting request: urn:request:84d31402-7de9-49ac-b1b5-fe81f4ad8edb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:22,071 [236] [DEBUG] [app] Ending request: urn:request:84d31402-7de9-49ac-b1b5-fe81f4ad8edb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:84d31402-7de9-49ac-b1b5-fe81f4ad8edb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:22,071 [233] [DEBUG] [app] Ending request: urn:request:64795ca4-cfaa-47ad-a48e-f7fc36f33a50 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64795ca4-cfaa-47ad-a48e-f7fc36f33a50', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:22,087 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:22,087 [236] [DEBUG] [app] Ending request: urn:request:c309e458-c0b4-4e8c-9400-d593492c4879 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c309e458-c0b4-4e8c-9400-d593492c4879', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:26:22,096 [231] [DEBUG] [app] Ending request: urn:request:fa3330eb-b133-47aa-83c9-0de4e0e0009c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fa3330eb-b133-47aa-83c9-0de4e0e0009c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:22,096 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:22,096 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.037) securityscanningnotificationworker stdout | 2023-01-17 05:26:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:26:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:37.256652+00:00 (in 13.719142 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:26:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:23 UTC)" (scheduled at 2023-01-17 05:26:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:26:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 26, 23, 537856), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:26:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 23, 537856), True, datetime.datetime(2023, 1, 17, 5, 26, 23, 537856), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:26:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 23, 537856), True, datetime.datetime(2023, 1, 17, 5, 26, 23, 537856), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 26, 23, 537856), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:26:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:26:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:35.646354+00:00 (in 9.999552 seconds) notificationworker stdout | 2023-01-17 05:26:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:35 UTC)" (scheduled at 2023-01-17 05:26:25.646354+00:00) notificationworker stdout | 2023-01-17 05:26:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 25, 647209), True, datetime.datetime(2023, 1, 17, 5, 26, 25, 647209), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:25,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:25,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:25,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:28,539 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:26:28,622 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:26:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:26:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:35.853396+00:00 (in 6.889255 seconds) exportactionlogsworker stdout | 2023-01-17 05:26:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:28 UTC)" (scheduled at 2023-01-17 05:26:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:26:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:26:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:26:29,546 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:26:30,014 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:26:30,032 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 05:26:30,249 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:30,261 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:30,261 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:30,271 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:26:30,375 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:26:30,522 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:26:30,988 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:26:30,990 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:26:30,993 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:26:30,995 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:26:30,997 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:26:33,644 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:26:33,917 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:26:34,970 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:26:34,972 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:26:34,974 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:26:34,989 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:26:35,025 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:26:35,218 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:26:35,281 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:26:35,283 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:26:35,285 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:26:35,288 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:26:35,295 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:26:35,297 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:26:35,301 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:26:35,303 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:26:35,306 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} notificationworker stdout | 2023-01-17 05:26:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:45.646354+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 05:26:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:45 UTC)" (scheduled at 2023-01-17 05:26:35.646354+00:00) notificationworker stdout | 2023-01-17 05:26:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 35, 647254), True, datetime.datetime(2023, 1, 17, 5, 26, 35, 647254), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:45 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:26:35,847 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} exportactionlogsworker stdout | 2023-01-17 05:26:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:26:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:54.963413+00:00 (in 19.109541 seconds) exportactionlogsworker stdout | 2023-01-17 05:26:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:35 UTC)" (scheduled at 2023-01-17 05:26:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:26:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:26:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 35, 854237), True, datetime.datetime(2023, 1, 17, 5, 26, 35, 854237), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:26:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:26:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:26:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:35 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:26:35,950 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:26:36,213 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:26:36,346 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:26:36,895 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:26:37,060 [233] [DEBUG] [app] Starting request: urn:request:a65d4e80-d074-4d4a-b6e4-0619c9601ddf (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:37,060 [231] [DEBUG] [app] Starting request: urn:request:5694135b-f593-4b89-ad9a-c8f6911f927b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:37,065 [230] [DEBUG] [app] Starting request: urn:request:016cb1de-a604-4601-8f8f-ab6fbc0647fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,065 [242] [DEBUG] [app] Starting request: urn:request:b2a015bb-2688-4d71-a823-bb2af3cdb998 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,066 [230] [DEBUG] [app] Ending request: urn:request:016cb1de-a604-4601-8f8f-ab6fbc0647fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:016cb1de-a604-4601-8f8f-ab6fbc0647fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,066 [242] [DEBUG] [app] Ending request: urn:request:b2a015bb-2688-4d71-a823-bb2af3cdb998 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2a015bb-2688-4d71-a823-bb2af3cdb998', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:26:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:26:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:37,070 [236] [DEBUG] [app] Starting request: urn:request:ca5e70f3-85e8-42e8-b81f-3db7479a2171 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:37,070 [231] [DEBUG] [app] Starting request: urn:request:1123572b-3db9-462b-888c-2b4b9372015a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:37,071 [236] [DEBUG] [app] Ending request: urn:request:ca5e70f3-85e8-42e8-b81f-3db7479a2171 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ca5e70f3-85e8-42e8-b81f-3db7479a2171', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:37,071 [231] [DEBUG] [app] Ending request: urn:request:1123572b-3db9-462b-888c-2b4b9372015a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1123572b-3db9-462b-888c-2b4b9372015a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:26:37,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:37,088 [233] [DEBUG] [app] Ending request: urn:request:a65d4e80-d074-4d4a-b6e4-0619c9601ddf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a65d4e80-d074-4d4a-b6e4-0619c9601ddf', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:37,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:37,088 [231] [DEBUG] [app] Ending request: urn:request:5694135b-f593-4b89-ad9a-c8f6911f927b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5694135b-f593-4b89-ad9a-c8f6911f927b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:37,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:26:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:26:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:26:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:37.537407+00:00 (in 0.280306 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:26:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:37 UTC)" (scheduled at 2023-01-17 05:26:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:26:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:26:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 37, 257536), True, datetime.datetime(2023, 1, 17, 5, 26, 37, 257536), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:26:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:26:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:26:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:26:37,414 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:26:37,450 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} securityscanningnotificationworker stdout | 2023-01-17 05:26:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:26:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:37.256652+00:00 (in 59.718686 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:26:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:37 UTC)" (scheduled at 2023-01-17 05:26:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:26:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:26:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:26:37,663 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:26:40,281 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:40,295 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:40,295 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:40,306 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:26:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:26:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:10.738568+00:00 (in 29.999539 seconds) buildlogsarchiver stdout | 2023-01-17 05:26:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:10 UTC)" (scheduled at 2023-01-17 05:26:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:26:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 26, 40, 739374), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:26:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:26:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:26:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:26:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:26:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:42.941536+00:00 (in 1.792827 seconds) namespacegcworker stdout | 2023-01-17 05:26:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:41 UTC)" (scheduled at 2023-01-17 05:26:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:26:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:26:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 41, 149162), True, datetime.datetime(2023, 1, 17, 5, 26, 41, 149162), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:26:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:26:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:26:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:26:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:26:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:06.941185+00:00 (in 23.999178 seconds) namespacegcworker stdout | 2023-01-17 05:26:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:42 UTC)" (scheduled at 2023-01-17 05:26:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:26:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:26:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:27:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:55.646354+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 05:26:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:55 UTC)" (scheduled at 2023-01-17 05:26:45.646354+00:00) notificationworker stdout | 2023-01-17 05:26:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 45, 647273), True, datetime.datetime(2023, 1, 17, 5, 26, 45, 647273), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:26:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:26:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:26:49.537472+00:00 (in 2.600009 seconds) securityworker stdout | 2023-01-17 05:26:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:16 UTC)" (scheduled at 2023-01-17 05:26:46.936988+00:00) securityworker stdout | 2023-01-17 05:26:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:26:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:26:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:16.936988+00:00 (in 27.399014 seconds) securityworker stdout | 2023-01-17 05:26:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:19 UTC)" (scheduled at 2023-01-17 05:26:49.537472+00:00) securityworker stdout | 2023-01-17 05:26:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:26:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:26:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:26:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:26:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:26:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:26:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:26:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:26:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:26:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:26:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:26:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:26:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:26:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:19 UTC)" executed successfully builder stdout | 2023-01-17 05:26:50,306 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:26:50,318 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:26:50,318 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:26:50,329 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:26:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:26:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:21.348605+00:00 (in 29.999418 seconds) gcworker stdout | 2023-01-17 05:26:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:21 UTC)" (scheduled at 2023-01-17 05:26:51.348605+00:00) gcworker stdout | 2023-01-17 05:26:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723611349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:26:51,368 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:26:51,368 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:26:51,368 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:26:52,060 [235] [DEBUG] [app] Starting request: urn:request:a96fc0ee-0116-4372-9f7d-ca11d88dfe7a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:52,060 [236] [DEBUG] [app] Starting request: urn:request:210e611e-009c-4365-925d-03719d47c2b2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:26:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:26:52,065 [242] [DEBUG] [app] Starting request: urn:request:6f684f0e-b6fe-4d47-9f00-006d83fe02f0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:52,065 [241] [DEBUG] [app] Starting request: urn:request:52134158-819a-4667-82ef-3396fffe1b05 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:26:52,065 [242] [DEBUG] [app] Ending request: urn:request:6f684f0e-b6fe-4d47-9f00-006d83fe02f0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f684f0e-b6fe-4d47-9f00-006d83fe02f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:52,066 [241] [DEBUG] [app] Ending request: urn:request:52134158-819a-4667-82ef-3396fffe1b05 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:52134158-819a-4667-82ef-3396fffe1b05', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:26:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:26:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:26:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:26:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:26:52,070 [231] [DEBUG] [app] Starting request: urn:request:80458fec-70e0-4f83-b3df-6f7935f0c01a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:52,070 [231] [DEBUG] [app] Ending request: urn:request:80458fec-70e0-4f83-b3df-6f7935f0c01a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:80458fec-70e0-4f83-b3df-6f7935f0c01a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:26:52,070 [233] [DEBUG] [app] Starting request: urn:request:d3ffa0ab-509d-4927-ba06-ef50b557a51f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:26:52,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:26:52,071 [233] [DEBUG] [app] Ending request: urn:request:d3ffa0ab-509d-4927-ba06-ef50b557a51f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3ffa0ab-509d-4927-ba06-ef50b557a51f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:26:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:26:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:26:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:26:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:26:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:26:52,087 [235] [DEBUG] [app] Ending request: urn:request:a96fc0ee-0116-4372-9f7d-ca11d88dfe7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a96fc0ee-0116-4372-9f7d-ca11d88dfe7a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:52,087 [236] [DEBUG] [app] Ending request: urn:request:210e611e-009c-4365-925d-03719d47c2b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:210e611e-009c-4365-925d-03719d47c2b2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:26:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:26:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:26:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:26:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) exportactionlogsworker stdout | 2023-01-17 05:26:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:26:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:28.963662+00:00 (in 33.999750 seconds) exportactionlogsworker stdout | 2023-01-17 05:26:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:54 UTC)" (scheduled at 2023-01-17 05:26:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:26:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 26, 54, 964361), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:26:54,976 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 54, 964361), True, datetime.datetime(2023, 1, 17, 5, 26, 54, 964361), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:26:54,980 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 26, 54, 964361), True, datetime.datetime(2023, 1, 17, 5, 26, 54, 964361), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 26, 54, 964361), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:26:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:26:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:31:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:26:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:01.636362+00:00 (in 5.989525 seconds) notificationworker stdout | 2023-01-17 05:26:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:05 UTC)" (scheduled at 2023-01-17 05:26:55.646354+00:00) notificationworker stdout | 2023-01-17 05:26:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:26:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 26, 55, 647264), True, datetime.datetime(2023, 1, 17, 5, 26, 55, 647264), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:26:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:26:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:26:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:26:58,565 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:26:58,657 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:26:59,582 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:27:00,040 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:27:00,053 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} builder stdout | 2023-01-17 05:27:00,339 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:00,351 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:00,351 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:00,361 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:27:00,396 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:27:00,551 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:27:00,995 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:27:00,997 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:27:01,000 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:27:01,003 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:27:01,005 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:27:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:05.646354+00:00 (in 4.009475 seconds) notificationworker stdout | 2023-01-17 05:27:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:01 UTC)" (scheduled at 2023-01-17 05:27:01.636362+00:00) notificationworker stdout | 2023-01-17 05:27:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:27:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:27:03,664 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:27:03,953 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:27:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:27:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:19.346120+00:00 (in 14.408851 seconds) repositorygcworker stdout | 2023-01-17 05:27:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:04 UTC)" (scheduled at 2023-01-17 05:27:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:27:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:27:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 4, 937710), True, datetime.datetime(2023, 1, 17, 5, 27, 4, 937710), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:27:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:27:04,948 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:27:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:27:04,977 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:27:04,979 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:27:04,982 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:27:05,015 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:27:05,060 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:27:05,242 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:27:05,288 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:27:05,290 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:27:05,293 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:27:05,296 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:27:05,306 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:27:05,310 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:27:05,312 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:27:05,314 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:27:05,317 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:27:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:15.646354+00:00 (in 9.999451 seconds) notificationworker stdout | 2023-01-17 05:27:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:15 UTC)" (scheduled at 2023-01-17 05:27:05.646354+00:00) notificationworker stdout | 2023-01-17 05:27:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 5, 647296), True, datetime.datetime(2023, 1, 17, 5, 27, 5, 647296), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:27:05,853 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:27:05,986 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:27:06,242 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:27:06,356 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:27:06,911 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} namespacegcworker stdout | 2023-01-17 05:27:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:27:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:41.148245+00:00 (in 34.206567 seconds) namespacegcworker stdout | 2023-01-17 05:27:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:32:06 UTC)" (scheduled at 2023-01-17 05:27:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:27:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 27, 6, 942090), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:27:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 27, 6, 942090), True, datetime.datetime(2023, 1, 17, 5, 27, 6, 942090), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:27:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 27, 6, 942090), True, datetime.datetime(2023, 1, 17, 5, 27, 6, 942090), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 27, 6, 942090), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:27:06,961 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:27:06,961 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:32:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:27:07,060 [231] [DEBUG] [app] Starting request: urn:request:66aa03c8-bfb0-412d-8d9d-8c6d796a2578 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:07,060 [233] [DEBUG] [app] Starting request: urn:request:ec13468a-c956-4e31-8795-4378d4efcd82 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:07,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:07,065 [230] [DEBUG] [app] Starting request: urn:request:45eff6c5-e1d2-4855-8c5b-5b4d51e509e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:07,065 [243] [DEBUG] [app] Starting request: urn:request:035737d6-18b9-4bd6-9c08-e1ea8618c74d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:07,066 [243] [DEBUG] [app] Ending request: urn:request:035737d6-18b9-4bd6-9c08-e1ea8618c74d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:035737d6-18b9-4bd6-9c08-e1ea8618c74d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:07,066 [230] [DEBUG] [app] Ending request: urn:request:45eff6c5-e1d2-4855-8c5b-5b4d51e509e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:45eff6c5-e1d2-4855-8c5b-5b4d51e509e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:27:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:27:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:07,070 [233] [DEBUG] [app] Starting request: urn:request:8dda36a6-a325-4eb1-9f43-5716d7e2de89 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:07,071 [233] [DEBUG] [app] Ending request: urn:request:8dda36a6-a325-4eb1-9f43-5716d7e2de89 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8dda36a6-a325-4eb1-9f43-5716d7e2de89', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:07,071 [236] [DEBUG] [app] Starting request: urn:request:7f012d23-41b2-44cc-a592-03d63d9ad0b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:07,071 [236] [DEBUG] [app] Ending request: urn:request:7f012d23-41b2-44cc-a592-03d63d9ad0b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f012d23-41b2-44cc-a592-03d63d9ad0b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:07,087 [233] [DEBUG] [app] Ending request: urn:request:ec13468a-c956-4e31-8795-4378d4efcd82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec13468a-c956-4e31-8795-4378d4efcd82', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:07,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:27:07,090 [231] [DEBUG] [app] Ending request: urn:request:66aa03c8-bfb0-412d-8d9d-8c6d796a2578 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66aa03c8-bfb0-412d-8d9d-8c6d796a2578', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:07,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:07,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) teamsyncworker stdout | 2023-01-17 05:27:07,420 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:27:07,478 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:27:07,685 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:27:10,367 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:10,379 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:10,379 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:10,389 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:27:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:27:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:40.738568+00:00 (in 29.999562 seconds) buildlogsarchiver stdout | 2023-01-17 05:27:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:40 UTC)" (scheduled at 2023-01-17 05:27:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:27:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 27, 10, 739383), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:27:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:27:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:27:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:25.646354+00:00 (in 9.999556 seconds) notificationworker stdout | 2023-01-17 05:27:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:25 UTC)" (scheduled at 2023-01-17 05:27:15.646354+00:00) notificationworker stdout | 2023-01-17 05:27:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 15, 647160), True, datetime.datetime(2023, 1, 17, 5, 27, 15, 647160), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:27:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:19.537472+00:00 (in 2.599972 seconds) securityworker stdout | 2023-01-17 05:27:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:46 UTC)" (scheduled at 2023-01-17 05:27:16.936988+00:00) securityworker stdout | 2023-01-17 05:27:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:27:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:27:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:04.936835+00:00 (in 45.590200 seconds) repositorygcworker stdout | 2023-01-17 05:27:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:19 UTC)" (scheduled at 2023-01-17 05:27:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:27:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:27:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:27:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:46.936988+00:00 (in 27.399014 seconds) securityworker stdout | 2023-01-17 05:27:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:49 UTC)" (scheduled at 2023-01-17 05:27:19.537472+00:00) securityworker stdout | 2023-01-17 05:27:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:27:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:27:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:19,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:19,556 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:19,556 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:27:19,556 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:27:19,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:27:19,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:19,558 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:27:19,558 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:49 UTC)" executed successfully builder stdout | 2023-01-17 05:27:20,397 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:20,409 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:20,409 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:20,419 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:27:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:27:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:51.348605+00:00 (in 29.999502 seconds) gcworker stdout | 2023-01-17 05:27:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:51 UTC)" (scheduled at 2023-01-17 05:27:21.348605+00:00) gcworker stdout | 2023-01-17 05:27:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723641349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:27:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:27:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:27:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:27:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:27:22,060 [236] [DEBUG] [app] Starting request: urn:request:510bb7d0-a47a-4304-924e-212efbc0ef47 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:22,060 [231] [DEBUG] [app] Starting request: urn:request:40dbdf1b-3a9f-438e-9321-ef60a3b6767c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:22,065 [230] [DEBUG] [app] Starting request: urn:request:72771b4c-4187-4bf9-a7b9-46defc2701af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:22,066 [242] [DEBUG] [app] Starting request: urn:request:00edd810-e6fe-49b0-9302-2162c40cc801 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:22,066 [230] [DEBUG] [app] Ending request: urn:request:72771b4c-4187-4bf9-a7b9-46defc2701af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:72771b4c-4187-4bf9-a7b9-46defc2701af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:22,066 [242] [DEBUG] [app] Ending request: urn:request:00edd810-e6fe-49b0-9302-2162c40cc801 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:00edd810-e6fe-49b0-9302-2162c40cc801', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:27:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:27:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:22,070 [231] [DEBUG] [app] Starting request: urn:request:64b5b360-105c-49e8-a658-f92e9ab3f90a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:22,070 [233] [DEBUG] [app] Starting request: urn:request:4c44e867-9df4-4933-8947-a00c00dea548 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:22,071 [231] [DEBUG] [app] Ending request: urn:request:64b5b360-105c-49e8-a658-f92e9ab3f90a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64b5b360-105c-49e8-a658-f92e9ab3f90a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:22,071 [233] [DEBUG] [app] Ending request: urn:request:4c44e867-9df4-4933-8947-a00c00dea548 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c44e867-9df4-4933-8947-a00c00dea548', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:27:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:22,088 [231] [DEBUG] [app] Ending request: urn:request:40dbdf1b-3a9f-438e-9321-ef60a3b6767c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:40dbdf1b-3a9f-438e-9321-ef60a3b6767c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:22,088 [236] [DEBUG] [app] Ending request: urn:request:510bb7d0-a47a-4304-924e-212efbc0ef47 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:510bb7d0-a47a-4304-924e-212efbc0ef47', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:27:22,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:27:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:27:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:35.646354+00:00 (in 9.999579 seconds) notificationworker stdout | 2023-01-17 05:27:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:35 UTC)" (scheduled at 2023-01-17 05:27:25.646354+00:00) notificationworker stdout | 2023-01-17 05:27:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 25, 647150), True, datetime.datetime(2023, 1, 17, 5, 27, 25, 647150), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:28,602 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:27:28,686 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:27:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:27:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:35.853396+00:00 (in 6.889212 seconds) exportactionlogsworker stdout | 2023-01-17 05:27:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:28 UTC)" (scheduled at 2023-01-17 05:27:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:27:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:27:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:27:29,607 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:27:30,071 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:27:30,088 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:27:30,401 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 05:27:30,430 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:30,442 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:30,442 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:30,453 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:30,566 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:27:31,002 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:27:31,005 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:27:31,007 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:27:31,009 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:27:31,012 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:27:33,700 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:27:33,959 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:27:34,984 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:27:34,987 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:27:34,989 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:27:35,036 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:27:35,072 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:27:35,248 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:27:35,295 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:27:35,300 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:27:35,303 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:27:35,306 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:27:35,312 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:27:35,318 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:27:35,320 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:27:35,323 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:27:35,326 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:27:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:45.646354+00:00 (in 9.999572 seconds) notificationworker stdout | 2023-01-17 05:27:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:45 UTC)" (scheduled at 2023-01-17 05:27:35.646354+00:00) notificationworker stdout | 2023-01-17 05:27:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 35, 647142), True, datetime.datetime(2023, 1, 17, 5, 27, 35, 647142), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:27:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:27:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:28.963662+00:00 (in 53.109729 seconds) exportactionlogsworker stdout | 2023-01-17 05:27:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:35 UTC)" (scheduled at 2023-01-17 05:27:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:27:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:27:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 35, 854341), True, datetime.datetime(2023, 1, 17, 5, 27, 35, 854341), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:27:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:27:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:27:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:27:35,880 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:27:36,006 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:27:36,277 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:27:36,391 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:27:36,947 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:27:37,060 [235] [DEBUG] [app] Starting request: urn:request:b91370f7-f4e3-4c8e-b7ae-de73080bc02a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:37,060 [231] [DEBUG] [app] Starting request: urn:request:a8bced3d-52fc-4f17-9523-51a451518c00 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:37,065 [243] [DEBUG] [app] Starting request: urn:request:c02d8f86-1124-4423-a66a-c38194fde3d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:37,065 [242] [DEBUG] [app] Starting request: urn:request:7cf3f967-fc73-44b3-981b-165f31c38098 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:37,065 [243] [DEBUG] [app] Ending request: urn:request:c02d8f86-1124-4423-a66a-c38194fde3d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c02d8f86-1124-4423-a66a-c38194fde3d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:27:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:27:37,066 [242] [DEBUG] [app] Ending request: urn:request:7cf3f967-fc73-44b3-981b-165f31c38098 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7cf3f967-fc73-44b3-981b-165f31c38098', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:27:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:27:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:27:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:37,070 [236] [DEBUG] [app] Starting request: urn:request:f7d7f6e5-fd64-4cec-9706-9bd23b121d69 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:37,070 [235] [DEBUG] [app] Starting request: urn:request:a94f80ae-a949-4109-af37-1a28010a4d56 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:37,070 [236] [DEBUG] [app] Ending request: urn:request:f7d7f6e5-fd64-4cec-9706-9bd23b121d69 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7d7f6e5-fd64-4cec-9706-9bd23b121d69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:37,070 [235] [DEBUG] [app] Ending request: urn:request:a94f80ae-a949-4109-af37-1a28010a4d56 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a94f80ae-a949-4109-af37-1a28010a4d56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:27:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:37,087 [235] [DEBUG] [app] Ending request: urn:request:b91370f7-f4e3-4c8e-b7ae-de73080bc02a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b91370f7-f4e3-4c8e-b7ae-de73080bc02a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:37,087 [231] [DEBUG] [app] Ending request: urn:request:a8bced3d-52fc-4f17-9523-51a451518c00 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8bced3d-52fc-4f17-9523-51a451518c00', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:27:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) securityscanningnotificationworker stdout | 2023-01-17 05:27:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:27:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:37.537407+00:00 (in 0.280309 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:27:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:37 UTC)" (scheduled at 2023-01-17 05:27:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:27:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:27:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 37, 257501), True, datetime.datetime(2023, 1, 17, 5, 27, 37, 257501), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:27:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:27:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:27:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:27:37,453 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:27:37,484 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} securityscanningnotificationworker stdout | 2023-01-17 05:27:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:27:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:37.256652+00:00 (in 59.718808 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:27:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:37 UTC)" (scheduled at 2023-01-17 05:27:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:27:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:27:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:27:37,721 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:27:40,463 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:40,475 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:40,475 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:40,486 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:27:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:27:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:10.738568+00:00 (in 29.999585 seconds) buildlogsarchiver stdout | 2023-01-17 05:27:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:10 UTC)" (scheduled at 2023-01-17 05:27:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:27:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 27, 40, 739363), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:27:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:27:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:27:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:27:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:27:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:42.941536+00:00 (in 1.792837 seconds) namespacegcworker stdout | 2023-01-17 05:27:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:41 UTC)" (scheduled at 2023-01-17 05:27:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:27:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:27:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 41, 149139), True, datetime.datetime(2023, 1, 17, 5, 27, 41, 149139), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:27:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:27:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:27:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:27:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:27:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:41.148245+00:00 (in 58.206230 seconds) namespacegcworker stdout | 2023-01-17 05:27:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:42 UTC)" (scheduled at 2023-01-17 05:27:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:27:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:27:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:28:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:55.646354+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 05:27:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:55 UTC)" (scheduled at 2023-01-17 05:27:45.646354+00:00) notificationworker stdout | 2023-01-17 05:27:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 45, 647179), True, datetime.datetime(2023, 1, 17, 5, 27, 45, 647179), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:27:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:27:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:27:49.537472+00:00 (in 2.600009 seconds) securityworker stdout | 2023-01-17 05:27:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:16 UTC)" (scheduled at 2023-01-17 05:27:46.936988+00:00) securityworker stdout | 2023-01-17 05:27:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:27:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:27:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:16.936988+00:00 (in 27.399032 seconds) securityworker stdout | 2023-01-17 05:27:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:19 UTC)" (scheduled at 2023-01-17 05:27:49.537472+00:00) securityworker stdout | 2023-01-17 05:27:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:27:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:27:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:27:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:27:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:27:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:27:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:27:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:27:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:27:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:27:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:27:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:27:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:27:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:19 UTC)" executed successfully builder stdout | 2023-01-17 05:27:50,493 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:27:50,505 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:27:50,505 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:27:50,516 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:27:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:27:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:21.348605+00:00 (in 29.999538 seconds) gcworker stdout | 2023-01-17 05:27:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:21 UTC)" (scheduled at 2023-01-17 05:27:51.348605+00:00) gcworker stdout | 2023-01-17 05:27:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723671349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:27:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:27:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:27:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:27:52,059 [231] [DEBUG] [app] Starting request: urn:request:7f468532-fa3e-4f2c-b4c7-487aa574f35d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:52,059 [233] [DEBUG] [app] Starting request: urn:request:5dc3abb1-b2ee-47bf-b345-8cce03bd4161 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:27:52,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,060 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:27:52,064 [242] [DEBUG] [app] Starting request: urn:request:b6bc8ee4-53b4-42de-8538-f882facce833 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:52,064 [243] [DEBUG] [app] Starting request: urn:request:5df0daa6-371b-42b8-be16-67a1904b43aa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:27:52,065 [242] [DEBUG] [app] Ending request: urn:request:b6bc8ee4-53b4-42de-8538-f882facce833 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b6bc8ee4-53b4-42de-8538-f882facce833', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:27:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:27:52,065 [243] [DEBUG] [app] Ending request: urn:request:5df0daa6-371b-42b8-be16-67a1904b43aa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5df0daa6-371b-42b8-be16-67a1904b43aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:27:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:27:52,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:52,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:27:52,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:27:52,069 [235] [DEBUG] [app] Starting request: urn:request:21288f88-8595-4bbb-991f-c70c55049c2d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:52,070 [235] [DEBUG] [app] Ending request: urn:request:21288f88-8595-4bbb-991f-c70c55049c2d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21288f88-8595-4bbb-991f-c70c55049c2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:52,070 [231] [DEBUG] [app] Starting request: urn:request:f664d94d-8b7b-483b-b96a-87c1aae6163d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:27:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:27:52,070 [231] [DEBUG] [app] Ending request: urn:request:f664d94d-8b7b-483b-b96a-87c1aae6163d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f664d94d-8b7b-483b-b96a-87c1aae6163d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:27:52,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:27:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:27:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:27:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:27:52,087 [231] [DEBUG] [app] Ending request: urn:request:7f468532-fa3e-4f2c-b4c7-487aa574f35d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f468532-fa3e-4f2c-b4c7-487aa574f35d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:52,087 [233] [DEBUG] [app] Ending request: urn:request:5dc3abb1-b2ee-47bf-b345-8cce03bd4161 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5dc3abb1-b2ee-47bf-b345-8cce03bd4161', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:27:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:52,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:27:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:27:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:27:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:27:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:27:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:01.636362+00:00 (in 5.989522 seconds) notificationworker stdout | 2023-01-17 05:27:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:05 UTC)" (scheduled at 2023-01-17 05:27:55.646354+00:00) notificationworker stdout | 2023-01-17 05:27:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:27:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 27, 55, 647248), True, datetime.datetime(2023, 1, 17, 5, 27, 55, 647248), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:27:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:27:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:27:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:27:58,610 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:27:58,713 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:27:59,643 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:28:00,098 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:28:00,109 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:28:00,437 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 05:28:00,526 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:00,538 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:00,538 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:00,549 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:00,602 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:28:01,010 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:28:01,012 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:28:01,015 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:28:01,017 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:28:01,019 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:28:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:05.646354+00:00 (in 4.009487 seconds) notificationworker stdout | 2023-01-17 05:28:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:01 UTC)" (scheduled at 2023-01-17 05:28:01.636362+00:00) notificationworker stdout | 2023-01-17 05:28:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:28:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:28:03,736 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:28:03,995 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:28:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:28:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:19.346120+00:00 (in 14.408780 seconds) repositorygcworker stdout | 2023-01-17 05:28:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:04 UTC)" (scheduled at 2023-01-17 05:28:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:28:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:28:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 4, 937739), True, datetime.datetime(2023, 1, 17, 5, 28, 4, 937739), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:28:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:28:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:28:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:28:04,992 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:28:04,994 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:28:04,997 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:28:05,070 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:28:05,086 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:28:05,270 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:28:05,306 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:28:05,310 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:28:05,312 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:28:05,315 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:28:05,322 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:28:05,326 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:28:05,329 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:28:05,332 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:28:05,334 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:28:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:15.646354+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:28:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:15 UTC)" (scheduled at 2023-01-17 05:28:05.646354+00:00) notificationworker stdout | 2023-01-17 05:28:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 5, 647192), True, datetime.datetime(2023, 1, 17, 5, 28, 5, 647192), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:28:05,886 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:28:06,041 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:28:06,287 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:28:06,397 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:28:06,976 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:28:07,059 [231] [DEBUG] [app] Starting request: urn:request:ed856d83-9f5f-4faf-bd16-efe567512bae (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:07,059 [235] [DEBUG] [app] Starting request: urn:request:640d7ec1-08f5-4b20-b86a-22f836ee1867 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:07,064 [242] [DEBUG] [app] Starting request: urn:request:4d42a6db-d86e-4e00-8815-48cbc09113ac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:07,065 [241] [DEBUG] [app] Starting request: urn:request:5c2a26c4-2754-40a6-9c08-3331503176bc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:07,065 [242] [DEBUG] [app] Ending request: urn:request:4d42a6db-d86e-4e00-8815-48cbc09113ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d42a6db-d86e-4e00-8815-48cbc09113ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:28:07,065 [241] [DEBUG] [app] Ending request: urn:request:5c2a26c4-2754-40a6-9c08-3331503176bc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c2a26c4-2754-40a6-9c08-3331503176bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:28:07,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:07,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:07,069 [236] [DEBUG] [app] Starting request: urn:request:7f12a394-917c-495a-8b04-4b2b3f5cc042 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:07,070 [233] [DEBUG] [app] Starting request: urn:request:062d569f-23c6-47c5-9f6d-8338ba6f2e4e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:07,070 [236] [DEBUG] [app] Ending request: urn:request:7f12a394-917c-495a-8b04-4b2b3f5cc042 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f12a394-917c-495a-8b04-4b2b3f5cc042', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:07,070 [233] [DEBUG] [app] Ending request: urn:request:062d569f-23c6-47c5-9f6d-8338ba6f2e4e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:062d569f-23c6-47c5-9f6d-8338ba6f2e4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:07,087 [231] [DEBUG] [app] Ending request: urn:request:ed856d83-9f5f-4faf-bd16-efe567512bae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed856d83-9f5f-4faf-bd16-efe567512bae', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:07,087 [235] [DEBUG] [app] Ending request: urn:request:640d7ec1-08f5-4b20-b86a-22f836ee1867 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:640d7ec1-08f5-4b20-b86a-22f836ee1867', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:28:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" teamsyncworker stdout | 2023-01-17 05:28:07,489 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:28:07,517 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:28:07,756 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:28:10,555 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:10,567 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:10,567 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:10,578 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:28:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:28:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:40.738568+00:00 (in 29.999531 seconds) buildlogsarchiver stdout | 2023-01-17 05:28:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:40 UTC)" (scheduled at 2023-01-17 05:28:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:28:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 28, 10, 739472), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:28:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:28:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:28:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:25.646354+00:00 (in 9.999587 seconds) notificationworker stdout | 2023-01-17 05:28:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:25 UTC)" (scheduled at 2023-01-17 05:28:15.646354+00:00) notificationworker stdout | 2023-01-17 05:28:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 15, 647131), True, datetime.datetime(2023, 1, 17, 5, 28, 15, 647131), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:28:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:19.537472+00:00 (in 2.599995 seconds) securityworker stdout | 2023-01-17 05:28:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:46 UTC)" (scheduled at 2023-01-17 05:28:16.936988+00:00) securityworker stdout | 2023-01-17 05:28:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:28:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:28:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:04.936835+00:00 (in 45.590280 seconds) repositorygcworker stdout | 2023-01-17 05:28:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:19 UTC)" (scheduled at 2023-01-17 05:28:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:28:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:28:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:28:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:46.936988+00:00 (in 27.399010 seconds) securityworker stdout | 2023-01-17 05:28:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:49 UTC)" (scheduled at 2023-01-17 05:28:19.537472+00:00) securityworker stdout | 2023-01-17 05:28:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:28:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:28:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:28:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:28:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:28:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:28:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:49 UTC)" executed successfully builder stdout | 2023-01-17 05:28:20,588 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:20,600 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:20,600 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:20,611 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:28:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:28:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:51.348605+00:00 (in 29.999548 seconds) gcworker stdout | 2023-01-17 05:28:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:51 UTC)" (scheduled at 2023-01-17 05:28:21.348605+00:00) gcworker stdout | 2023-01-17 05:28:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723701349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:28:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:28:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:28:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:28:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:28:22,059 [231] [DEBUG] [app] Starting request: urn:request:fd37fdea-3f4c-4584-88ca-73d2eb95ac22 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:22,060 [235] [DEBUG] [app] Starting request: urn:request:04a2b42c-8713-4177-b527-8b8c9732ae31 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:22,064 [242] [DEBUG] [app] Starting request: urn:request:06e211a9-f1b8-4e47-8e28-9452b6863163 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:22,065 [230] [DEBUG] [app] Starting request: urn:request:c0af8318-d7ab-4189-bf66-2a4201a50d2f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:22,065 [242] [DEBUG] [app] Ending request: urn:request:06e211a9-f1b8-4e47-8e28-9452b6863163 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:06e211a9-f1b8-4e47-8e28-9452b6863163', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:22,065 [230] [DEBUG] [app] Ending request: urn:request:c0af8318-d7ab-4189-bf66-2a4201a50d2f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c0af8318-d7ab-4189-bf66-2a4201a50d2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:28:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:28:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:22,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:22,070 [236] [DEBUG] [app] Starting request: urn:request:5c1c5571-b556-45ea-93c1-d362c010ce0a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:22,070 [235] [DEBUG] [app] Starting request: urn:request:0dc08bbf-c508-4143-8f16-47511970653e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:22,071 [236] [DEBUG] [app] Ending request: urn:request:5c1c5571-b556-45ea-93c1-d362c010ce0a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5c1c5571-b556-45ea-93c1-d362c010ce0a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:22,071 [235] [DEBUG] [app] Ending request: urn:request:0dc08bbf-c508-4143-8f16-47511970653e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0dc08bbf-c508-4143-8f16-47511970653e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:22,088 [235] [DEBUG] [app] Ending request: urn:request:04a2b42c-8713-4177-b527-8b8c9732ae31 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04a2b42c-8713-4177-b527-8b8c9732ae31', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:28:22,089 [231] [DEBUG] [app] Ending request: urn:request:fd37fdea-3f4c-4584-88ca-73d2eb95ac22 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd37fdea-3f4c-4584-88ca-73d2eb95ac22', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:22,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 05:28:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:28:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:35.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:28:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:35 UTC)" (scheduled at 2023-01-17 05:28:25.646354+00:00) notificationworker stdout | 2023-01-17 05:28:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 25, 647235), True, datetime.datetime(2023, 1, 17, 5, 28, 25, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:28,624 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:28:28,749 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:28:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:28:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:35.853396+00:00 (in 6.889246 seconds) exportactionlogsworker stdout | 2023-01-17 05:28:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:28 UTC)" (scheduled at 2023-01-17 05:28:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:28:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:28:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:28:29,651 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:28:30,129 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:28:30,144 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:28:30,473 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 05:28:30,621 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue securityworker stdout | 2023-01-17 05:28:30,628 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:28:30,633 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:30,634 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:30,644 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:31,016 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:28:31,018 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:28:31,021 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:28:31,024 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:28:31,027 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:28:33,772 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:28:34,007 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:28:35,000 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:28:35,002 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:28:35,005 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:28:35,082 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:28:35,094 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 05:28:35,306 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:28:35,318 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:28:35,320 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:28:35,323 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:28:35,325 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:28:35,330 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:28:35,334 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:28:35,336 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:28:35,341 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:28:35,343 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:28:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:36.635997+00:00 (in 0.989179 seconds) notificationworker stdout | 2023-01-17 05:28:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:45 UTC)" (scheduled at 2023-01-17 05:28:35.646354+00:00) notificationworker stdout | 2023-01-17 05:28:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 35, 647229), True, datetime.datetime(2023, 1, 17, 5, 28, 35, 647229), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:28:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:28:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:28.963662+00:00 (in 53.109780 seconds) exportactionlogsworker stdout | 2023-01-17 05:28:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:35 UTC)" (scheduled at 2023-01-17 05:28:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:28:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:28:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 35, 854235), True, datetime.datetime(2023, 1, 17, 5, 28, 35, 854235), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:28:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:28:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:28:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:28:35,921 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:28:36,053 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:28:36,294 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:28:36,422 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:28:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:45.646354+00:00 (in 9.009898 seconds) notificationworker stdout | 2023-01-17 05:28:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:33:36 UTC)" (scheduled at 2023-01-17 05:28:36.635997+00:00) notificationworker stdout | 2023-01-17 05:28:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 28, 36, 636811), 'notification/%']) notificationworker stdout | 2023-01-17 05:28:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 28, 36, 636811), True, datetime.datetime(2023, 1, 17, 5, 28, 36, 636811), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:28:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 28, 36, 636811), True, datetime.datetime(2023, 1, 17, 5, 28, 36, 636811), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 28, 36, 636811), 'notification/%']) notificationworker stdout | 2023-01-17 05:28:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:33:36 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:28:36,997 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:28:37,060 [231] [DEBUG] [app] Starting request: urn:request:26b544e4-d96d-4b1b-a8f2-2cffdd12a994 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:37,060 [233] [DEBUG] [app] Starting request: urn:request:e8074730-0657-4a3b-936b-5f624419b5fa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:37,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:37,065 [242] [DEBUG] [app] Starting request: urn:request:168a02f7-a8b6-4d72-b949-5e99c82ff21b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:37,065 [241] [DEBUG] [app] Starting request: urn:request:fea91e3c-b84b-4028-a977-d24c600e7265 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:37,066 [242] [DEBUG] [app] Ending request: urn:request:168a02f7-a8b6-4d72-b949-5e99c82ff21b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:168a02f7-a8b6-4d72-b949-5e99c82ff21b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:28:37,066 [241] [DEBUG] [app] Ending request: urn:request:fea91e3c-b84b-4028-a977-d24c600e7265 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fea91e3c-b84b-4028-a977-d24c600e7265', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:28:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:37,071 [235] [DEBUG] [app] Starting request: urn:request:2f5a6a45-0c33-4e58-a2f0-29f061067eb8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:37,071 [236] [DEBUG] [app] Starting request: urn:request:d10d8047-1671-4741-aa79-701eb5e5e7d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:37,071 [235] [DEBUG] [app] Ending request: urn:request:2f5a6a45-0c33-4e58-a2f0-29f061067eb8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f5a6a45-0c33-4e58-a2f0-29f061067eb8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:37,072 [236] [DEBUG] [app] Ending request: urn:request:d10d8047-1671-4741-aa79-701eb5e5e7d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d10d8047-1671-4741-aa79-701eb5e5e7d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:37,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:28:37,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:37,088 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:37,091 [231] [DEBUG] [app] Ending request: urn:request:26b544e4-d96d-4b1b-a8f2-2cffdd12a994 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26b544e4-d96d-4b1b-a8f2-2cffdd12a994', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:37,091 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) gunicorn-web stdout | 2023-01-17 05:28:37,091 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:28:37,097 [233] [DEBUG] [app] Ending request: urn:request:e8074730-0657-4a3b-936b-5f624419b5fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e8074730-0657-4a3b-936b-5f624419b5fa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:37,097 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:37,097 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.038 47 0.038) securityscanningnotificationworker stdout | 2023-01-17 05:28:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:28:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:37.537407+00:00 (in 0.280337 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:28:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:37 UTC)" (scheduled at 2023-01-17 05:28:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:28:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:28:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 37, 257509), True, datetime.datetime(2023, 1, 17, 5, 28, 37, 257509), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:28:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:28:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:28:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:28:37,522 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:28:37,530 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} securityscanningnotificationworker stdout | 2023-01-17 05:28:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:28:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:37.256652+00:00 (in 59.718785 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:28:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:37 UTC)" (scheduled at 2023-01-17 05:28:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:28:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:28:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:37 UTC)" executed successfully blobuploadcleanupworker stdout | 2023-01-17 05:28:37,782 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 05:28:40,654 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:40,667 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:40,667 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:40,678 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:28:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:28:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:10.738568+00:00 (in 29.999531 seconds) buildlogsarchiver stdout | 2023-01-17 05:28:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:10 UTC)" (scheduled at 2023-01-17 05:28:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:28:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 28, 40, 739463), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:28:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:28:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:28:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:28:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:28:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:42.941536+00:00 (in 1.792803 seconds) namespacegcworker stdout | 2023-01-17 05:28:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:41 UTC)" (scheduled at 2023-01-17 05:28:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:28:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:28:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 41, 149139), True, datetime.datetime(2023, 1, 17, 5, 28, 41, 149139), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:28:41,167 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:28:41,167 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:28:41,167 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:28:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:28:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:41.148245+00:00 (in 58.206225 seconds) namespacegcworker stdout | 2023-01-17 05:28:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:42 UTC)" (scheduled at 2023-01-17 05:28:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:28:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:28:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:29:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:55.646354+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 05:28:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:55 UTC)" (scheduled at 2023-01-17 05:28:45.646354+00:00) notificationworker stdout | 2023-01-17 05:28:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 45, 647239), True, datetime.datetime(2023, 1, 17, 5, 28, 45, 647239), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:28:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:28:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:28:49.537472+00:00 (in 2.600015 seconds) securityworker stdout | 2023-01-17 05:28:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:16 UTC)" (scheduled at 2023-01-17 05:28:46.936988+00:00) securityworker stdout | 2023-01-17 05:28:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:28:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:28:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:16.936988+00:00 (in 27.399043 seconds) securityworker stdout | 2023-01-17 05:28:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:19 UTC)" (scheduled at 2023-01-17 05:28:49.537472+00:00) securityworker stdout | 2023-01-17 05:28:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:28:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:28:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:28:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:28:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:28:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:28:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:28:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:28:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:28:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:28:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:28:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:28:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:28:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:19 UTC)" executed successfully builder stdout | 2023-01-17 05:28:50,685 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:28:50,698 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:28:50,698 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:28:50,708 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:28:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:28:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:21.348605+00:00 (in 29.999513 seconds) gcworker stdout | 2023-01-17 05:28:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:21 UTC)" (scheduled at 2023-01-17 05:28:51.348605+00:00) gcworker stdout | 2023-01-17 05:28:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723731349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:28:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:28:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:28:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:28:52,060 [231] [DEBUG] [app] Starting request: urn:request:d5c9349a-1cd4-4549-a37a-8a1a73d2eab8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:52,060 [236] [DEBUG] [app] Starting request: urn:request:ffc91dcc-3171-4e84-8e48-fa18da3c68d4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:28:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:28:52,065 [242] [DEBUG] [app] Starting request: urn:request:bafba3bc-cf6a-43b9-bb4d-93091819282d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:52,066 [241] [DEBUG] [app] Starting request: urn:request:7fcab491-479a-4231-89de-007ede6444ac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:28:52,066 [242] [DEBUG] [app] Ending request: urn:request:bafba3bc-cf6a-43b9-bb4d-93091819282d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bafba3bc-cf6a-43b9-bb4d-93091819282d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:28:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:28:52,066 [241] [DEBUG] [app] Ending request: urn:request:7fcab491-479a-4231-89de-007ede6444ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7fcab491-479a-4231-89de-007ede6444ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:28:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:28:52,067 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:52,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:28:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:28:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:28:52,071 [235] [DEBUG] [app] Starting request: urn:request:b98920a9-c4e5-42a2-b354-8b93254af7ac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:52,071 [235] [DEBUG] [app] Ending request: urn:request:b98920a9-c4e5-42a2-b354-8b93254af7ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b98920a9-c4e5-42a2-b354-8b93254af7ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:52,071 [233] [DEBUG] [app] Starting request: urn:request:68e9dfab-a8ae-4d47-8fe9-f3e56ada04c6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:28:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:52,072 [233] [DEBUG] [app] Ending request: urn:request:68e9dfab-a8ae-4d47-8fe9-f3e56ada04c6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68e9dfab-a8ae-4d47-8fe9-f3e56ada04c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:28:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:28:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:28:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:28:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:28:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:28:52,088 [231] [DEBUG] [app] Ending request: urn:request:d5c9349a-1cd4-4549-a37a-8a1a73d2eab8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d5c9349a-1cd4-4549-a37a-8a1a73d2eab8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:52,088 [236] [DEBUG] [app] Ending request: urn:request:ffc91dcc-3171-4e84-8e48-fa18da3c68d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ffc91dcc-3171-4e84-8e48-fa18da3c68d4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:28:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:28:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:28:52,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:28:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:28:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:28:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:01.636362+00:00 (in 5.989521 seconds) notificationworker stdout | 2023-01-17 05:28:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:05 UTC)" (scheduled at 2023-01-17 05:28:55.646354+00:00) notificationworker stdout | 2023-01-17 05:28:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:28:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 28, 55, 647233), True, datetime.datetime(2023, 1, 17, 5, 28, 55, 647233), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:28:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:28:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:28:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:28:58,660 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:28:58,782 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:28:59,659 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:29:00,150 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:29:00,164 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:29:00,503 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:29:00,649 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:29:00,719 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:00,731 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:00,731 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:00,741 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:01,023 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:29:01,026 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:29:01,029 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:29:01,031 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:29:01,034 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:29:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:05.646354+00:00 (in 4.009496 seconds) notificationworker stdout | 2023-01-17 05:29:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:01 UTC)" (scheduled at 2023-01-17 05:29:01.636362+00:00) notificationworker stdout | 2023-01-17 05:29:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:29:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:29:03,808 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:29:04,042 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:29:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:29:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:19.346120+00:00 (in 14.408821 seconds) repositorygcworker stdout | 2023-01-17 05:29:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:04 UTC)" (scheduled at 2023-01-17 05:29:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:29:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:29:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 4, 937770), True, datetime.datetime(2023, 1, 17, 5, 29, 4, 937770), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:29:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:29:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:29:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:29:05,007 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:29:05,010 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:29:05,013 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:29:05,117 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:29:05,128 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:29:05,327 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:29:05,329 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:29:05,332 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:29:05,334 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:29:05,337 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} repositorygcworker stdout | 2023-01-17 05:29:05,340 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-registry stdout | 2023-01-17 05:29:05,343 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:29:05,345 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:29:05,350 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:29:05,352 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:29:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:15.646354+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 05:29:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:15 UTC)" (scheduled at 2023-01-17 05:29:05.646354+00:00) notificationworker stdout | 2023-01-17 05:29:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 5, 647231), True, datetime.datetime(2023, 1, 17, 5, 29, 5, 647231), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:29:05,954 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:29:06,089 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:29:06,311 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:29:06,428 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:29:07,030 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:29:07,060 [231] [DEBUG] [app] Starting request: urn:request:5aa6f2c1-cb91-4402-9bdd-21e5b218c3bb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:07,060 [236] [DEBUG] [app] Starting request: urn:request:e349ecf2-720e-469e-a5f5-d0cf5244b2ec (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:07,065 [230] [DEBUG] [app] Starting request: urn:request:17011b22-30fb-4b0b-84b9-5fa795b61ea2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:07,065 [243] [DEBUG] [app] Starting request: urn:request:4c61f327-6497-4fd8-a8e9-7c9ca298564c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:07,065 [230] [DEBUG] [app] Ending request: urn:request:17011b22-30fb-4b0b-84b9-5fa795b61ea2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:17011b22-30fb-4b0b-84b9-5fa795b61ea2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:29:07,066 [243] [DEBUG] [app] Ending request: urn:request:4c61f327-6497-4fd8-a8e9-7c9ca298564c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c61f327-6497-4fd8-a8e9-7c9ca298564c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:29:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:29:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:07,070 [235] [DEBUG] [app] Starting request: urn:request:4c0d21ec-3d91-4c79-a6d2-a35b1bc9869f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:07,070 [231] [DEBUG] [app] Starting request: urn:request:5a7bccb5-a5b8-4e46-aafd-f7d9a1b06681 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:07,071 [235] [DEBUG] [app] Ending request: urn:request:4c0d21ec-3d91-4c79-a6d2-a35b1bc9869f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c0d21ec-3d91-4c79-a6d2-a35b1bc9869f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:07,071 [231] [DEBUG] [app] Ending request: urn:request:5a7bccb5-a5b8-4e46-aafd-f7d9a1b06681 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a7bccb5-a5b8-4e46-aafd-f7d9a1b06681', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:29:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:07,088 [231] [DEBUG] [app] Ending request: urn:request:5aa6f2c1-cb91-4402-9bdd-21e5b218c3bb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5aa6f2c1-cb91-4402-9bdd-21e5b218c3bb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:07,088 [236] [DEBUG] [app] Ending request: urn:request:e349ecf2-720e-469e-a5f5-d0cf5244b2ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e349ecf2-720e-469e-a5f5-d0cf5244b2ec', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:29:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) teamsyncworker stdout | 2023-01-17 05:29:07,549 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:29:07,564 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:29:07,817 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:29:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:29:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:40.738568+00:00 (in 29.999491 seconds) buildlogsarchiver stdout | 2023-01-17 05:29:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:40 UTC)" (scheduled at 2023-01-17 05:29:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:29:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 29, 10, 739492), False, 50, 1, 0]) builder stdout | 2023-01-17 05:29:10,749 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue buildlogsarchiver stdout | 2023-01-17 05:29:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:29:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:29:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:40 UTC)" executed successfully builder stdout | 2023-01-17 05:29:10,760 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:10,760 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:10,771 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:25.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 05:29:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:25 UTC)" (scheduled at 2023-01-17 05:29:15.646354+00:00) notificationworker stdout | 2023-01-17 05:29:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 15, 647198), True, datetime.datetime(2023, 1, 17, 5, 29, 15, 647198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:29:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:19.537472+00:00 (in 2.599989 seconds) securityworker stdout | 2023-01-17 05:29:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:46 UTC)" (scheduled at 2023-01-17 05:29:16.936988+00:00) securityworker stdout | 2023-01-17 05:29:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:16,953 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:16,956 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:16,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:29:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:29:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:04.936835+00:00 (in 45.590197 seconds) repositorygcworker stdout | 2023-01-17 05:29:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:19 UTC)" (scheduled at 2023-01-17 05:29:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:29:19,347 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:29:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:29:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:46.936988+00:00 (in 27.399057 seconds) securityworker stdout | 2023-01-17 05:29:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:49 UTC)" (scheduled at 2023-01-17 05:29:19.537472+00:00) securityworker stdout | 2023-01-17 05:29:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:29:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:29:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:29:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:29:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:29:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:29:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:49 UTC)" executed successfully builder stdout | 2023-01-17 05:29:20,781 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:20,793 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:20,793 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:20,804 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:29:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:29:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:51.348605+00:00 (in 29.999498 seconds) gcworker stdout | 2023-01-17 05:29:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:51 UTC)" (scheduled at 2023-01-17 05:29:21.348605+00:00) gcworker stdout | 2023-01-17 05:29:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723761349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:29:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:29:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:29:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:29:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:29:22,060 [235] [DEBUG] [app] Starting request: urn:request:55342127-a1a1-459e-b73f-31da29e1dd94 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:22,060 [236] [DEBUG] [app] Starting request: urn:request:7cbcd292-0eec-4de5-a14b-92d99ca17e3f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:22,065 [243] [DEBUG] [app] Starting request: urn:request:abc85d31-2aa2-44e8-8ab1-1a8481487924 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:22,065 [230] [DEBUG] [app] Starting request: urn:request:cb1189a5-ad5a-4ad7-b03d-db2d22355e54 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:22,065 [243] [DEBUG] [app] Ending request: urn:request:abc85d31-2aa2-44e8-8ab1-1a8481487924 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abc85d31-2aa2-44e8-8ab1-1a8481487924', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:29:22,066 [230] [DEBUG] [app] Ending request: urn:request:cb1189a5-ad5a-4ad7-b03d-db2d22355e54 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb1189a5-ad5a-4ad7-b03d-db2d22355e54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:29:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:29:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:22,070 [231] [DEBUG] [app] Starting request: urn:request:7316ef61-dd0d-4c01-bea5-6bc0f0088989 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:22,070 [235] [DEBUG] [app] Starting request: urn:request:9b0d7fe8-7c84-401c-9f48-c5e655ad0ab8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:22,070 [231] [DEBUG] [app] Ending request: urn:request:7316ef61-dd0d-4c01-bea5-6bc0f0088989 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7316ef61-dd0d-4c01-bea5-6bc0f0088989', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:22,071 [235] [DEBUG] [app] Ending request: urn:request:9b0d7fe8-7c84-401c-9f48-c5e655ad0ab8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b0d7fe8-7c84-401c-9f48-c5e655ad0ab8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:22,088 [236] [DEBUG] [app] Ending request: urn:request:7cbcd292-0eec-4de5-a14b-92d99ca17e3f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7cbcd292-0eec-4de5-a14b-92d99ca17e3f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:29:22,093 [235] [DEBUG] [app] Ending request: urn:request:55342127-a1a1-459e-b73f-31da29e1dd94 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55342127-a1a1-459e-b73f-31da29e1dd94', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:22,093 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:22,094 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) notificationworker stdout | 2023-01-17 05:29:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:35.646354+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 05:29:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:35 UTC)" (scheduled at 2023-01-17 05:29:25.646354+00:00) notificationworker stdout | 2023-01-17 05:29:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 25, 647210), True, datetime.datetime(2023, 1, 17, 5, 29, 25, 647210), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:28,671 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:29:28,812 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:29:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:29:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:35.853396+00:00 (in 6.889335 seconds) exportactionlogsworker stdout | 2023-01-17 05:29:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:28 UTC)" (scheduled at 2023-01-17 05:29:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:29:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:29:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:29:29,686 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:29:30,183 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:29:30,199 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:29:30,538 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:29:30,684 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:29:30,806 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:30,819 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:30,819 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:30,830 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:31,030 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:29:31,033 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:29:31,035 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:29:31,038 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:29:31,040 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:29:33,844 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:29:34,061 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:29:35,015 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:29:35,017 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:29:35,019 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:29:35,131 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:29:35,138 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:29:35,336 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:29:35,338 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:29:35,342 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:29:35,344 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:29:35,346 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:29:35,349 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:29:35,352 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:29:35,358 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:29:35,360 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:29:35,371 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:29:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:45.646354+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 05:29:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:45 UTC)" (scheduled at 2023-01-17 05:29:35.646354+00:00) notificationworker stdout | 2023-01-17 05:29:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 35, 647218), True, datetime.datetime(2023, 1, 17, 5, 29, 35, 647218), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:29:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:29:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:28.963662+00:00 (in 53.109818 seconds) exportactionlogsworker stdout | 2023-01-17 05:29:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:35 UTC)" (scheduled at 2023-01-17 05:29:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:29:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:29:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 35, 854211), True, datetime.datetime(2023, 1, 17, 5, 29, 35, 854211), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:29:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:29:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:29:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:29:35,989 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:29:36,124 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:29:36,342 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:29:36,453 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:29:37,055 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:29:37,060 [235] [DEBUG] [app] Starting request: urn:request:7234b676-c5d5-410e-bfc2-7eba536a2524 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:37,060 [236] [DEBUG] [app] Starting request: urn:request:ed9e2de1-9fc6-4d7b-9d59-9aca181d85d5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:37,065 [242] [DEBUG] [app] Starting request: urn:request:2df354e5-228d-498d-9206-e52154a61d6c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:37,065 [230] [DEBUG] [app] Starting request: urn:request:402de932-f50a-4db0-a0bf-c5e0c08f9e67 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:37,065 [242] [DEBUG] [app] Ending request: urn:request:2df354e5-228d-498d-9206-e52154a61d6c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2df354e5-228d-498d-9206-e52154a61d6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:37,065 [230] [DEBUG] [app] Ending request: urn:request:402de932-f50a-4db0-a0bf-c5e0c08f9e67 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:402de932-f50a-4db0-a0bf-c5e0c08f9e67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:29:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:29:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:29:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:37,070 [233] [DEBUG] [app] Starting request: urn:request:5e50dfbf-01cd-438c-ac16-549d0e99b867 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:37,070 [231] [DEBUG] [app] Starting request: urn:request:3dd36856-4736-4de0-9a38-8ee3fa40011d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:37,070 [233] [DEBUG] [app] Ending request: urn:request:5e50dfbf-01cd-438c-ac16-549d0e99b867 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5e50dfbf-01cd-438c-ac16-549d0e99b867', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:37,071 [231] [DEBUG] [app] Ending request: urn:request:3dd36856-4736-4de0-9a38-8ee3fa40011d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3dd36856-4736-4de0-9a38-8ee3fa40011d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:29:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:37,082 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:37,087 [235] [DEBUG] [app] Ending request: urn:request:7234b676-c5d5-410e-bfc2-7eba536a2524 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7234b676-c5d5-410e-bfc2-7eba536a2524', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:29:37,091 [236] [DEBUG] [app] Ending request: urn:request:ed9e2de1-9fc6-4d7b-9d59-9aca181d85d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed9e2de1-9fc6-4d7b-9d59-9aca181d85d5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:37,091 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.033 47 0.032) gunicorn-web stdout | 2023-01-17 05:29:37,092 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:29:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:29:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:37.537407+00:00 (in 0.280323 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:29:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:37 UTC)" (scheduled at 2023-01-17 05:29:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:29:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:29:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 37, 257489), True, datetime.datetime(2023, 1, 17, 5, 29, 37, 257489), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:29:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:29:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:29:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:29:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:29:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:37.256652+00:00 (in 59.718865 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:29:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:37 UTC)" (scheduled at 2023-01-17 05:29:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:29:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:29:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:29:37,562 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:29:37,600 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:29:37,853 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:29:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:29:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:10.738568+00:00 (in 29.999523 seconds) buildlogsarchiver stdout | 2023-01-17 05:29:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:10 UTC)" (scheduled at 2023-01-17 05:29:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:29:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 29, 40, 739426), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:29:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:29:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:29:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:10 UTC)" executed successfully builder stdout | 2023-01-17 05:29:40,839 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:40,851 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:40,852 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:40,862 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:29:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:29:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:42.941536+00:00 (in 1.792781 seconds) namespacegcworker stdout | 2023-01-17 05:29:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:41 UTC)" (scheduled at 2023-01-17 05:29:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:29:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:29:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 41, 149174), True, datetime.datetime(2023, 1, 17, 5, 29, 41, 149174), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:29:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:29:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:29:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:29:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:29:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:41.148245+00:00 (in 58.206225 seconds) namespacegcworker stdout | 2023-01-17 05:29:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:42 UTC)" (scheduled at 2023-01-17 05:29:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:29:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:29:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:30:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:55.646354+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 05:29:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:55 UTC)" (scheduled at 2023-01-17 05:29:45.646354+00:00) notificationworker stdout | 2023-01-17 05:29:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 45, 647262), True, datetime.datetime(2023, 1, 17, 5, 29, 45, 647262), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:29:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:29:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:29:49.537472+00:00 (in 2.600014 seconds) securityworker stdout | 2023-01-17 05:29:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:16 UTC)" (scheduled at 2023-01-17 05:29:46.936988+00:00) securityworker stdout | 2023-01-17 05:29:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:29:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:29:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:16.936988+00:00 (in 27.399018 seconds) securityworker stdout | 2023-01-17 05:29:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:19 UTC)" (scheduled at 2023-01-17 05:29:49.537472+00:00) securityworker stdout | 2023-01-17 05:29:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:29:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:29:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:29:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:29:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:29:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:29:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:29:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:29:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:29:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:29:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:29:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:29:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:29:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:19 UTC)" executed successfully builder stdout | 2023-01-17 05:29:50,863 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:29:50,875 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:29:50,876 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:29:50,886 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:29:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:29:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:21.348605+00:00 (in 29.999502 seconds) gcworker stdout | 2023-01-17 05:29:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:21 UTC)" (scheduled at 2023-01-17 05:29:51.348605+00:00) gcworker stdout | 2023-01-17 05:29:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723791349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:29:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:29:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:29:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:29:52,060 [236] [DEBUG] [app] Starting request: urn:request:7af71adf-ea85-436e-a384-b6aefccbb19b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:52,060 [235] [DEBUG] [app] Starting request: urn:request:a29ee786-624e-4fe0-8bf4-0102f497d4d1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:29:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:52,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:29:52,065 [242] [DEBUG] [app] Starting request: urn:request:c6c1dd0a-7b3d-458d-b7ae-8c7aee6bbaa3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:52,065 [230] [DEBUG] [app] Starting request: urn:request:c8561b01-4136-4f85-9af9-1b5ca49a7400 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:29:52,066 [242] [DEBUG] [app] Ending request: urn:request:c6c1dd0a-7b3d-458d-b7ae-8c7aee6bbaa3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c6c1dd0a-7b3d-458d-b7ae-8c7aee6bbaa3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:29:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:29:52,066 [230] [DEBUG] [app] Ending request: urn:request:c8561b01-4136-4f85-9af9-1b5ca49a7400 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c8561b01-4136-4f85-9af9-1b5ca49a7400', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:29:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:29:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:29:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:29:52,071 [231] [DEBUG] [app] Starting request: urn:request:0534cb5f-7c11-4eee-b143-90dd3c381c98 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:52,071 [235] [DEBUG] [app] Starting request: urn:request:5cfea873-c6b3-4652-9821-7f936717cc10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:29:52,071 [231] [DEBUG] [app] Ending request: urn:request:0534cb5f-7c11-4eee-b143-90dd3c381c98 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0534cb5f-7c11-4eee-b143-90dd3c381c98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:52,071 [235] [DEBUG] [app] Ending request: urn:request:5cfea873-c6b3-4652-9821-7f936717cc10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5cfea873-c6b3-4652-9821-7f936717cc10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:29:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:29:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:29:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:29:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:29:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:29:52,088 [235] [DEBUG] [app] Ending request: urn:request:a29ee786-624e-4fe0-8bf4-0102f497d4d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a29ee786-624e-4fe0-8bf4-0102f497d4d1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:52,088 [236] [DEBUG] [app] Ending request: urn:request:7af71adf-ea85-436e-a384-b6aefccbb19b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7af71adf-ea85-436e-a384-b6aefccbb19b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:29:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:29:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:29:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:29:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:29:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:29:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:01.636362+00:00 (in 5.989520 seconds) notificationworker stdout | 2023-01-17 05:29:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:05 UTC)" (scheduled at 2023-01-17 05:29:55.646354+00:00) notificationworker stdout | 2023-01-17 05:29:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:29:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 29, 55, 647241), True, datetime.datetime(2023, 1, 17, 5, 29, 55, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:29:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:29:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:29:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:29:58,707 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:29:58,833 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:29:59,710 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:30:00,195 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:30:00,233 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:30:00,573 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:30:00,720 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:30:00,896 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:00,909 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:00,909 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:00,919 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:01,038 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:30:01,041 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:30:01,043 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:30:01,046 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:30:01,048 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:30:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:05.646354+00:00 (in 4.009489 seconds) notificationworker stdout | 2023-01-17 05:30:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:01 UTC)" (scheduled at 2023-01-17 05:30:01.636362+00:00) notificationworker stdout | 2023-01-17 05:30:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:30:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:30:03,878 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:30:04,066 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:30:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:30:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:19.346120+00:00 (in 14.408826 seconds) repositorygcworker stdout | 2023-01-17 05:30:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:04 UTC)" (scheduled at 2023-01-17 05:30:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:30:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:30:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 4, 937754), True, datetime.datetime(2023, 1, 17, 5, 30, 4, 937754), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:30:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:30:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:30:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:30:05,022 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:30:05,024 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:30:05,026 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} servicekey stdout | 2023-01-17 05:30:05,148 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 05:30:05,173 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:30:05,343 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:30:05,345 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:30:05,349 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:30:05,352 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:30:05,355 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:30:05,357 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:30:05,359 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:30:05,365 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:30:05,367 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:30:05,383 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:30:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:15.646354+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:30:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:15 UTC)" (scheduled at 2023-01-17 05:30:05.646354+00:00) notificationworker stdout | 2023-01-17 05:30:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 5, 647293), True, datetime.datetime(2023, 1, 17, 5, 30, 5, 647293), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:30:06,009 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:30:06,149 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:30:06,356 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:30:06,487 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:30:07,060 [231] [DEBUG] [app] Starting request: urn:request:50c85973-8df3-462b-bfb7-218f9cc7b546 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:07,060 [236] [DEBUG] [app] Starting request: urn:request:a8934ffb-9ac2-45c8-987a-39fcef89c2d2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:07,065 [243] [DEBUG] [app] Starting request: urn:request:4530ae43-5fe7-4704-944d-dbefac43c792 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:07,065 [242] [DEBUG] [app] Starting request: urn:request:1cd8d9a6-4709-4349-91b8-3333ef4e34ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:07,065 [243] [DEBUG] [app] Ending request: urn:request:4530ae43-5fe7-4704-944d-dbefac43c792 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4530ae43-5fe7-4704-944d-dbefac43c792', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:30:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:30:07,066 [242] [DEBUG] [app] Ending request: urn:request:1cd8d9a6-4709-4349-91b8-3333ef4e34ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1cd8d9a6-4709-4349-91b8-3333ef4e34ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:30:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:07,071 [235] [DEBUG] [app] Starting request: urn:request:db0b7e30-7443-4558-8ec1-c69e132a61f8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:07,071 [236] [DEBUG] [app] Starting request: urn:request:05384a16-8e00-47ea-af4a-0b5379a4c6fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:07,071 [235] [DEBUG] [app] Ending request: urn:request:db0b7e30-7443-4558-8ec1-c69e132a61f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db0b7e30-7443-4558-8ec1-c69e132a61f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:07,071 [236] [DEBUG] [app] Ending request: urn:request:05384a16-8e00-47ea-af4a-0b5379a4c6fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05384a16-8e00-47ea-af4a-0b5379a4c6fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) globalpromstats stdout | 2023-01-17 05:30:07,091 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} gunicorn-web stdout | 2023-01-17 05:30:07,098 [236] [DEBUG] [app] Ending request: urn:request:a8934ffb-9ac2-45c8-987a-39fcef89c2d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8934ffb-9ac2-45c8-987a-39fcef89c2d2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:07,098 [231] [DEBUG] [app] Ending request: urn:request:50c85973-8df3-462b-bfb7-218f9cc7b546 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:50c85973-8df3-462b-bfb7-218f9cc7b546', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:07,098 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:07,098 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:07,098 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:30:07,098 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.039 47 0.039) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.039 47 0.039) teamsyncworker stdout | 2023-01-17 05:30:07,584 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:30:07,611 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:30:07,888 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:30:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:30:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:40.738568+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2023-01-17 05:30:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:40 UTC)" (scheduled at 2023-01-17 05:30:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:30:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 30, 10, 739474), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:30:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:30:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:30:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:40 UTC)" executed successfully builder stdout | 2023-01-17 05:30:10,930 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:10,942 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:10,942 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:10,952 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:25.646354+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 05:30:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:25 UTC)" (scheduled at 2023-01-17 05:30:15.646354+00:00) notificationworker stdout | 2023-01-17 05:30:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 15, 647259), True, datetime.datetime(2023, 1, 17, 5, 30, 15, 647259), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:19.537472+00:00 (in 2.600007 seconds) securityworker stdout | 2023-01-17 05:30:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:46 UTC)" (scheduled at 2023-01-17 05:30:16.936988+00:00) securityworker stdout | 2023-01-17 05:30:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:30:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:30:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:04.936835+00:00 (in 45.590248 seconds) repositorygcworker stdout | 2023-01-17 05:30:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:19 UTC)" (scheduled at 2023-01-17 05:30:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:30:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:30:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:46.936988+00:00 (in 27.399015 seconds) securityworker stdout | 2023-01-17 05:30:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:49 UTC)" (scheduled at 2023-01-17 05:30:19.537472+00:00) securityworker stdout | 2023-01-17 05:30:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:30:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:30:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:30:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:30:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:30:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:30:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:49 UTC)" executed successfully builder stdout | 2023-01-17 05:30:20,953 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:20,965 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:20,965 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:20,976 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:30:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:30:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:51.348605+00:00 (in 29.999502 seconds) gcworker stdout | 2023-01-17 05:30:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:51 UTC)" (scheduled at 2023-01-17 05:30:21.348605+00:00) gcworker stdout | 2023-01-17 05:30:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723821349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:30:21,367 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:30:21,367 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:30:21,367 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:30:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:30:22,059 [235] [DEBUG] [app] Starting request: urn:request:1c91be86-2117-4cc6-a971-b1f5bb705e45 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:22,060 [236] [DEBUG] [app] Starting request: urn:request:43cfacb5-9f9f-4d96-8226-04848b0cc5a9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:22,065 [230] [DEBUG] [app] Starting request: urn:request:bb64b1fb-fb78-45bb-bed5-f6f3431c3c48 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:22,065 [243] [DEBUG] [app] Starting request: urn:request:f5849bbe-e185-47f3-9eef-aa73d4cf3f45 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:22,066 [230] [DEBUG] [app] Ending request: urn:request:bb64b1fb-fb78-45bb-bed5-f6f3431c3c48 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bb64b1fb-fb78-45bb-bed5-f6f3431c3c48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:30:22,066 [243] [DEBUG] [app] Ending request: urn:request:f5849bbe-e185-47f3-9eef-aa73d4cf3f45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5849bbe-e185-47f3-9eef-aa73d4cf3f45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:30:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:30:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:30:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:22,070 [236] [DEBUG] [app] Starting request: urn:request:c570114d-f465-4caf-b0d4-442636fc05d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:22,070 [231] [DEBUG] [app] Starting request: urn:request:fb904fd0-92f3-4f84-abeb-b4c1d2f4dcd4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:22,071 [236] [DEBUG] [app] Ending request: urn:request:c570114d-f465-4caf-b0d4-442636fc05d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c570114d-f465-4caf-b0d4-442636fc05d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:22,071 [231] [DEBUG] [app] Ending request: urn:request:fb904fd0-92f3-4f84-abeb-b4c1d2f4dcd4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fb904fd0-92f3-4f84-abeb-b4c1d2f4dcd4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:30:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:22,087 [236] [DEBUG] [app] Ending request: urn:request:43cfacb5-9f9f-4d96-8226-04848b0cc5a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43cfacb5-9f9f-4d96-8226-04848b0cc5a9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:30:22,088 [235] [DEBUG] [app] Ending request: urn:request:1c91be86-2117-4cc6-a971-b1f5bb705e45 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c91be86-2117-4cc6-a971-b1f5bb705e45', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:22,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:30:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) notificationworker stdout | 2023-01-17 05:30:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:35.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:30:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:35 UTC)" (scheduled at 2023-01-17 05:30:25.646354+00:00) notificationworker stdout | 2023-01-17 05:30:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 25, 647211), True, datetime.datetime(2023, 1, 17, 5, 30, 25, 647211), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:30:28,730 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:30:28,866 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:30:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:30:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:35.853396+00:00 (in 6.889271 seconds) exportactionlogsworker stdout | 2023-01-17 05:30:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:28 UTC)" (scheduled at 2023-01-17 05:30:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:30:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:30:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:30:29,739 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:30:30,231 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:30:30,263 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:30:30,598 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:30:30,742 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:30:30,987 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:30,999 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:30,999 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:31,009 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:31,045 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:30:31,048 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:30:31,050 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:30:31,053 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:30:31,056 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} expiredappspecifictokenworker stdout | 2023-01-17 05:30:33,902 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:30:34,094 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:30:35,029 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:30:35,032 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:30:35,034 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:30:35,183 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:30:35,186 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:30:35,351 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:30:35,353 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:30:35,356 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:30:35,359 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:30:35,361 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:30:35,364 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:30:35,367 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:30:35,372 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:30:35,374 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:30:35,397 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:30:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:45.646354+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 05:30:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:45 UTC)" (scheduled at 2023-01-17 05:30:35.646354+00:00) notificationworker stdout | 2023-01-17 05:30:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 35, 647360), True, datetime.datetime(2023, 1, 17, 5, 30, 35, 647360), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:30:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:30:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:28.963662+00:00 (in 53.109718 seconds) exportactionlogsworker stdout | 2023-01-17 05:30:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:35 UTC)" (scheduled at 2023-01-17 05:30:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:30:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:30:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 35, 854430), True, datetime.datetime(2023, 1, 17, 5, 30, 35, 854430), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:30:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:30:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:30:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:30:36,015 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:30:36,159 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:30:36,374 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:30:36,497 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:30:37,060 [231] [DEBUG] [app] Starting request: urn:request:18243c12-8ca4-4726-8c51-6b7873408582 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:37,060 [236] [DEBUG] [app] Starting request: urn:request:1c8faa69-878e-4fb1-92e2-2eb7cbe64c33 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:37,065 [230] [DEBUG] [app] Starting request: urn:request:d8a7266c-1626-4d14-8fe6-b426efa32672 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:37,066 [239] [DEBUG] [app] Starting request: urn:request:8963c34a-8701-4dca-80f0-340a7807a2b5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:37,066 [230] [DEBUG] [app] Ending request: urn:request:d8a7266c-1626-4d14-8fe6-b426efa32672 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8a7266c-1626-4d14-8fe6-b426efa32672', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.003) gunicorn-registry stdout | 2023-01-17 05:30:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:30:37,066 [239] [DEBUG] [app] Ending request: urn:request:8963c34a-8701-4dca-80f0-340a7807a2b5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8963c34a-8701-4dca-80f0-340a7807a2b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:30:37,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:30:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:37,070 [235] [DEBUG] [app] Starting request: urn:request:6c775e4c-aa70-4261-9f68-985ad662393d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:37,070 [231] [DEBUG] [app] Starting request: urn:request:0d20e35d-1cf4-454a-a54b-b9a7698f977b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:37,071 [235] [DEBUG] [app] Ending request: urn:request:6c775e4c-aa70-4261-9f68-985ad662393d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6c775e4c-aa70-4261-9f68-985ad662393d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:37,071 [231] [DEBUG] [app] Ending request: urn:request:0d20e35d-1cf4-454a-a54b-b9a7698f977b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d20e35d-1cf4-454a-a54b-b9a7698f977b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:30:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:37,088 [236] [DEBUG] [app] Ending request: urn:request:1c8faa69-878e-4fb1-92e2-2eb7cbe64c33 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c8faa69-878e-4fb1-92e2-2eb7cbe64c33', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:37,088 [231] [DEBUG] [app] Ending request: urn:request:18243c12-8ca4-4726-8c51-6b7873408582 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:18243c12-8ca4-4726-8c51-6b7873408582', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:30:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) globalpromstats stdout | 2023-01-17 05:30:37,100 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:30:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:30:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:37.537407+00:00 (in 0.280356 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:30:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:37 UTC)" (scheduled at 2023-01-17 05:30:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:30:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:30:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 37, 257535), True, datetime.datetime(2023, 1, 17, 5, 30, 37, 257535), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:30:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:30:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:30:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:30:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:30:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:23.537048+00:00 (in 45.999249 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:30:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:37 UTC)" (scheduled at 2023-01-17 05:30:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:30:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:30:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:30:37,601 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:30:37,618 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:30:37,923 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:30:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:30:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:10.738568+00:00 (in 29.999538 seconds) buildlogsarchiver stdout | 2023-01-17 05:30:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:10 UTC)" (scheduled at 2023-01-17 05:30:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:30:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 30, 40, 739443), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:30:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:30:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:30:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:10 UTC)" executed successfully builder stdout | 2023-01-17 05:30:41,020 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:41,032 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:41,032 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:41,043 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:30:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:30:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:42.941536+00:00 (in 1.792778 seconds) namespacegcworker stdout | 2023-01-17 05:30:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:41 UTC)" (scheduled at 2023-01-17 05:30:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:30:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:30:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 41, 149159), True, datetime.datetime(2023, 1, 17, 5, 30, 41, 149159), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:30:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:30:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:30:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:30:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:30:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:41.148245+00:00 (in 58.206237 seconds) namespacegcworker stdout | 2023-01-17 05:30:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:42 UTC)" (scheduled at 2023-01-17 05:30:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:30:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:30:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:31:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:30:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:55.646354+00:00 (in 9.999491 seconds) notificationworker stdout | 2023-01-17 05:30:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:55 UTC)" (scheduled at 2023-01-17 05:30:45.646354+00:00) notificationworker stdout | 2023-01-17 05:30:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 45, 647277), True, datetime.datetime(2023, 1, 17, 5, 30, 45, 647277), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:30:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:30:49.537472+00:00 (in 2.600024 seconds) securityworker stdout | 2023-01-17 05:30:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:16 UTC)" (scheduled at 2023-01-17 05:30:46.936988+00:00) securityworker stdout | 2023-01-17 05:30:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:30:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:30:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:16.936988+00:00 (in 27.399088 seconds) securityworker stdout | 2023-01-17 05:30:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:19 UTC)" (scheduled at 2023-01-17 05:30:49.537472+00:00) securityworker stdout | 2023-01-17 05:30:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:30:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:30:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:49,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:30:49,540 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:30:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:30:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:30:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:30:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:30:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:30:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:30:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:30:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:30:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:30:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:19 UTC)" executed successfully builder stdout | 2023-01-17 05:30:51,053 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:30:51,066 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:30:51,066 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:30:51,076 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:30:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:30:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:21.348605+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 05:30:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:21 UTC)" (scheduled at 2023-01-17 05:30:51.348605+00:00) gcworker stdout | 2023-01-17 05:30:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723851349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:30:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:30:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:30:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:30:52,060 [233] [DEBUG] [app] Starting request: urn:request:2589a334-0d65-4e78-90b1-3f849149c7fd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:52,060 [236] [DEBUG] [app] Starting request: urn:request:c616ad18-1399-4be6-b484-1b065813f496 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:30:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:52,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:30:52,065 [242] [DEBUG] [app] Starting request: urn:request:8815f2b8-6191-444d-81be-c45d1d51054b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:52,066 [242] [DEBUG] [app] Ending request: urn:request:8815f2b8-6191-444d-81be-c45d1d51054b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8815f2b8-6191-444d-81be-c45d1d51054b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:30:52,066 [230] [DEBUG] [app] Starting request: urn:request:6b9768c3-6347-4b68-8dee-47c6243a37e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:30:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:30:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:30:52,066 [230] [DEBUG] [app] Ending request: urn:request:6b9768c3-6347-4b68-8dee-47c6243a37e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b9768c3-6347-4b68-8dee-47c6243a37e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:30:52,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:52,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,069 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:30:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:52,070 [231] [DEBUG] [app] Starting request: urn:request:a66fffa8-8190-4e01-84c7-ba90d2aa1d4d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:52,071 [231] [DEBUG] [app] Ending request: urn:request:a66fffa8-8190-4e01-84c7-ba90d2aa1d4d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a66fffa8-8190-4e01-84c7-ba90d2aa1d4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:30:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:30:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:52,071 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:30:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:52,072 [235] [DEBUG] [app] Starting request: urn:request:28c23033-2be3-483b-9e48-2f799cf3f480 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:30:52,072 [235] [DEBUG] [app] Ending request: urn:request:28c23033-2be3-483b-9e48-2f799cf3f480 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28c23033-2be3-483b-9e48-2f799cf3f480', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:30:52,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:30:52,073 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:30:52,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:30:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:52,085 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:30:52,088 [236] [DEBUG] [app] Ending request: urn:request:c616ad18-1399-4be6-b484-1b065813f496 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c616ad18-1399-4be6-b484-1b065813f496', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:30:52,094 [233] [DEBUG] [app] Ending request: urn:request:2589a334-0d65-4e78-90b1-3f849149c7fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2589a334-0d65-4e78-90b1-3f849149c7fd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:30:52,094 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:30:52,094 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:30:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.035) notificationworker stdout | 2023-01-17 05:30:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:30:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:01.636362+00:00 (in 5.989517 seconds) notificationworker stdout | 2023-01-17 05:30:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:05 UTC)" (scheduled at 2023-01-17 05:30:55.646354+00:00) notificationworker stdout | 2023-01-17 05:30:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:30:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 30, 55, 647329), True, datetime.datetime(2023, 1, 17, 5, 30, 55, 647329), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:30:55,660 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:30:55,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:30:55,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:30:58,746 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:30:58,903 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:30:59,775 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:31:00,241 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:31:00,298 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:31:00,633 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:31:00,752 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:31:01,052 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:31:01,056 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:31:01,058 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:31:01,061 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:31:01,063 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:31:01,083 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:01,095 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:01,095 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:01,106 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:05.646354+00:00 (in 4.009453 seconds) notificationworker stdout | 2023-01-17 05:31:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:01 UTC)" (scheduled at 2023-01-17 05:31:01.636362+00:00) notificationworker stdout | 2023-01-17 05:31:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:31:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:31:03,938 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:31:04,129 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:31:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:31:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:17.345601+00:00 (in 12.408339 seconds) repositorygcworker stdout | 2023-01-17 05:31:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:04 UTC)" (scheduled at 2023-01-17 05:31:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:31:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:31:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 4, 937713), True, datetime.datetime(2023, 1, 17, 5, 31, 4, 937713), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:31:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:31:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:31:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:31:05,037 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:31:05,040 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:31:05,043 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:31:05,204 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:31:05,220 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:31:05,358 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:31:05,360 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:31:05,363 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:31:05,365 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:31:05,371 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:31:05,374 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:31:05,377 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:31:05,379 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:31:05,384 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:31:05,429 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:31:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:15.646354+00:00 (in 9.999436 seconds) notificationworker stdout | 2023-01-17 05:31:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:15 UTC)" (scheduled at 2023-01-17 05:31:05.646354+00:00) notificationworker stdout | 2023-01-17 05:31:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 5, 647436), True, datetime.datetime(2023, 1, 17, 5, 31, 5, 647436), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:31:06,051 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:31:06,182 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:31:06,393 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:31:06,504 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:31:07,059 [235] [DEBUG] [app] Starting request: urn:request:b1ac967e-4c8e-4a22-9a6d-5ee9a1cfc727 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:07,059 [233] [DEBUG] [app] Starting request: urn:request:3cdc3a3f-e6a6-4760-a1a2-8559f5dcf6b9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:07,065 [242] [DEBUG] [app] Starting request: urn:request:ef27744a-a089-4b1f-8218-98741c76c0ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,065 [240] [DEBUG] [app] Starting request: urn:request:2f2363d8-fde7-4b11-8191-b7ab418ff193 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,065 [242] [DEBUG] [app] Ending request: urn:request:ef27744a-a089-4b1f-8218-98741c76c0ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef27744a-a089-4b1f-8218-98741c76c0ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,065 [240] [DEBUG] [app] Ending request: urn:request:2f2363d8-fde7-4b11-8191-b7ab418ff193 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f2363d8-fde7-4b11-8191-b7ab418ff193', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:31:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:31:07,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:07,070 [231] [DEBUG] [app] Starting request: urn:request:492d5ae5-83c8-4ace-995d-2cbebc8eeb2c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:07,070 [231] [DEBUG] [app] Ending request: urn:request:492d5ae5-83c8-4ace-995d-2cbebc8eeb2c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:492d5ae5-83c8-4ace-995d-2cbebc8eeb2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:07,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:07,071 [236] [DEBUG] [app] Starting request: urn:request:c5e9b06d-ee65-43e2-9796-c240c54ba186 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:07,071 [236] [DEBUG] [app] Ending request: urn:request:c5e9b06d-ee65-43e2-9796-c240c54ba186 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5e9b06d-ee65-43e2-9796-c240c54ba186', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:07,087 [233] [DEBUG] [app] Ending request: urn:request:3cdc3a3f-e6a6-4760-a1a2-8559f5dcf6b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3cdc3a3f-e6a6-4760-a1a2-8559f5dcf6b9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:31:07,088 [235] [DEBUG] [app] Ending request: urn:request:b1ac967e-4c8e-4a22-9a6d-5ee9a1cfc727 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b1ac967e-4c8e-4a22-9a6d-5ee9a1cfc727', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:31:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:07,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) globalpromstats stdout | 2023-01-17 05:31:07,114 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:31:07,637 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:31:07,652 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:31:07,934 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:31:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:31:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:40.738568+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 05:31:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:40 UTC)" (scheduled at 2023-01-17 05:31:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:31:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 31, 10, 739440), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:31:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:31:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:31:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:40 UTC)" executed successfully builder stdout | 2023-01-17 05:31:11,116 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:11,128 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:11,128 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:11,139 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:25.646354+00:00 (in 9.999472 seconds) notificationworker stdout | 2023-01-17 05:31:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:25 UTC)" (scheduled at 2023-01-17 05:31:15.646354+00:00) notificationworker stdout | 2023-01-17 05:31:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 15, 647331), True, datetime.datetime(2023, 1, 17, 5, 31, 15, 647331), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:19.537472+00:00 (in 2.600041 seconds) securityworker stdout | 2023-01-17 05:31:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:46 UTC)" (scheduled at 2023-01-17 05:31:16.936988+00:00) securityworker stdout | 2023-01-17 05:31:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:31:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:31:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:19.346120+00:00 (in 2.000035 seconds) repositorygcworker stdout | 2023-01-17 05:31:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:17 UTC)" (scheduled at 2023-01-17 05:31:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:31:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 31, 17, 346474), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:31:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 17, 346474), True, datetime.datetime(2023, 1, 17, 5, 31, 17, 346474), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:31:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 17, 346474), True, datetime.datetime(2023, 1, 17, 5, 31, 17, 346474), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 31, 17, 346474), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:31:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:31:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:31:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:31:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:04.936835+00:00 (in 45.590253 seconds) repositorygcworker stdout | 2023-01-17 05:31:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:19 UTC)" (scheduled at 2023-01-17 05:31:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:31:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:31:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:46.936988+00:00 (in 27.399030 seconds) securityworker stdout | 2023-01-17 05:31:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:49 UTC)" (scheduled at 2023-01-17 05:31:19.537472+00:00) securityworker stdout | 2023-01-17 05:31:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:31:19,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:31:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:31:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:31:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:31:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:31:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:49 UTC)" executed successfully builder stdout | 2023-01-17 05:31:21,149 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:21,162 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:21,162 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:21,172 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:31:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:31:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:51.348605+00:00 (in 29.999508 seconds) gcworker stdout | 2023-01-17 05:31:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:51 UTC)" (scheduled at 2023-01-17 05:31:21.348605+00:00) gcworker stdout | 2023-01-17 05:31:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723881349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:31:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:31:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:31:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:31:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:31:22,060 [231] [DEBUG] [app] Starting request: urn:request:37926504-bba0-4b2d-91cd-cd0f5ee9b873 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:22,060 [236] [DEBUG] [app] Starting request: urn:request:a62481da-3916-4e73-8528-6f1722823e1f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:22,065 [242] [DEBUG] [app] Starting request: urn:request:a5e02239-0c00-4a08-a6fe-a5f9eb9cabeb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:22,065 [241] [DEBUG] [app] Starting request: urn:request:8a72e2cd-6c3f-4fc0-8c5b-7fe74fa9e459 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:22,065 [242] [DEBUG] [app] Ending request: urn:request:a5e02239-0c00-4a08-a6fe-a5f9eb9cabeb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5e02239-0c00-4a08-a6fe-a5f9eb9cabeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:31:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:31:22,066 [241] [DEBUG] [app] Ending request: urn:request:8a72e2cd-6c3f-4fc0-8c5b-7fe74fa9e459 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a72e2cd-6c3f-4fc0-8c5b-7fe74fa9e459', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:31:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:31:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:22,071 [235] [DEBUG] [app] Starting request: urn:request:a0b24686-1c77-429b-9208-1b32707c9701 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:22,071 [233] [DEBUG] [app] Starting request: urn:request:fe40c657-34df-48c9-a4a3-a107c81868d5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:22,071 [233] [DEBUG] [app] Ending request: urn:request:fe40c657-34df-48c9-a4a3-a107c81868d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe40c657-34df-48c9-a4a3-a107c81868d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:22,071 [235] [DEBUG] [app] Ending request: urn:request:a0b24686-1c77-429b-9208-1b32707c9701 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0b24686-1c77-429b-9208-1b32707c9701', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:22,088 [236] [DEBUG] [app] Ending request: urn:request:a62481da-3916-4e73-8528-6f1722823e1f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a62481da-3916-4e73-8528-6f1722823e1f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:22,089 [231] [DEBUG] [app] Ending request: urn:request:37926504-bba0-4b2d-91cd-cd0f5ee9b873 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37926504-bba0-4b2d-91cd-cd0f5ee9b873', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:31:22,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:31:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:31:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:31:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:37.256652+00:00 (in 13.719190 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:31:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:23 UTC)" (scheduled at 2023-01-17 05:31:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:31:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 31, 23, 537823), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:31:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 23, 537823), True, datetime.datetime(2023, 1, 17, 5, 31, 23, 537823), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:31:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 23, 537823), True, datetime.datetime(2023, 1, 17, 5, 31, 23, 537823), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 31, 23, 537823), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:31:23,555 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:31:23,555 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:35.646354+00:00 (in 9.999524 seconds) notificationworker stdout | 2023-01-17 05:31:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:35 UTC)" (scheduled at 2023-01-17 05:31:25.646354+00:00) notificationworker stdout | 2023-01-17 05:31:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 25, 647241), True, datetime.datetime(2023, 1, 17, 5, 31, 25, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:28,782 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:31:28,925 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} exportactionlogsworker stdout | 2023-01-17 05:31:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:31:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:35.853396+00:00 (in 6.889272 seconds) exportactionlogsworker stdout | 2023-01-17 05:31:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:28 UTC)" (scheduled at 2023-01-17 05:31:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:31:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:31:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:28 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 05:31:29,811 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:31:30,251 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:31:30,311 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:31:30,646 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:31:30,783 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:31:31,060 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:31:31,062 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:31:31,065 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:31:31,067 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:31:31,069 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:31:31,178 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:31,191 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:31,191 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:31,202 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:31:33,955 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:31:34,150 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:31:35,045 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:31:35,047 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:31:35,050 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:31:35,218 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:31:35,252 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:31:35,366 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:31:35,369 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:31:35,372 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:31:35,375 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:31:35,381 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:31:35,383 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:31:35,385 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:31:35,388 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:31:35,393 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:31:35,464 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:31:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:45.646354+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 05:31:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:45 UTC)" (scheduled at 2023-01-17 05:31:35.646354+00:00) notificationworker stdout | 2023-01-17 05:31:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 35, 647347), True, datetime.datetime(2023, 1, 17, 5, 31, 35, 647347), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:31:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:31:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:54.963413+00:00 (in 19.109511 seconds) exportactionlogsworker stdout | 2023-01-17 05:31:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:35 UTC)" (scheduled at 2023-01-17 05:31:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:31:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:31:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 35, 854366), True, datetime.datetime(2023, 1, 17, 5, 31, 35, 854366), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:31:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:31:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:31:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:31:36,084 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:31:36,216 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:31:36,415 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:31:36,539 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:31:37,059 [235] [DEBUG] [app] Starting request: urn:request:2fa858f1-ab6e-41a1-9ccb-d5e03b141c6c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:37,059 [231] [DEBUG] [app] Starting request: urn:request:4c2b0f38-81a7-411c-8510-e3136a27ac94 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:37,064 [230] [DEBUG] [app] Starting request: urn:request:df740619-3227-4f7f-89fd-c97f4730064b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:37,065 [242] [DEBUG] [app] Starting request: urn:request:fa9d43e6-5f95-4856-8e60-dcf327d6c0eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:37,065 [230] [DEBUG] [app] Ending request: urn:request:df740619-3227-4f7f-89fd-c97f4730064b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df740619-3227-4f7f-89fd-c97f4730064b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:31:37,065 [242] [DEBUG] [app] Ending request: urn:request:fa9d43e6-5f95-4856-8e60-dcf327d6c0eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fa9d43e6-5f95-4856-8e60-dcf327d6c0eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:31:37,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:31:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:37,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:37,070 [236] [DEBUG] [app] Starting request: urn:request:3d2048a1-f5e4-4e6f-abc1-65f859086668 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:37,070 [233] [DEBUG] [app] Starting request: urn:request:93077de1-becd-4d03-9b90-04b99d7f4283 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:37,070 [236] [DEBUG] [app] Ending request: urn:request:3d2048a1-f5e4-4e6f-abc1-65f859086668 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d2048a1-f5e4-4e6f-abc1-65f859086668', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:37,070 [233] [DEBUG] [app] Ending request: urn:request:93077de1-becd-4d03-9b90-04b99d7f4283 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93077de1-becd-4d03-9b90-04b99d7f4283', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:37,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:37,086 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:37,088 [231] [DEBUG] [app] Ending request: urn:request:4c2b0f38-81a7-411c-8510-e3136a27ac94 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c2b0f38-81a7-411c-8510-e3136a27ac94', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:31:37,096 [235] [DEBUG] [app] Ending request: urn:request:2fa858f1-ab6e-41a1-9ccb-d5e03b141c6c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2fa858f1-ab6e-41a1-9ccb-d5e03b141c6c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:37,096 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:37,096 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.038 47 0.038) globalpromstats stdout | 2023-01-17 05:31:37,151 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:31:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:31:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:37.537407+00:00 (in 0.280321 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:31:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:37 UTC)" (scheduled at 2023-01-17 05:31:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:31:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:31:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 37, 257506), True, datetime.datetime(2023, 1, 17, 5, 31, 37, 257506), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:31:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:31:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:31:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:31:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:31:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:37.256652+00:00 (in 59.718898 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:31:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:37 UTC)" (scheduled at 2023-01-17 05:31:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:31:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:31:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:31:37,673 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:31:37,687 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:31:37,969 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:31:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:31:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:10.738568+00:00 (in 29.999542 seconds) buildlogsarchiver stdout | 2023-01-17 05:31:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:10 UTC)" (scheduled at 2023-01-17 05:31:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:31:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 31, 40, 739457), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:31:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:31:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:31:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:31:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:31:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:42.941536+00:00 (in 1.792824 seconds) namespacegcworker stdout | 2023-01-17 05:31:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:41 UTC)" (scheduled at 2023-01-17 05:31:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:31:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:31:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 41, 149167), True, datetime.datetime(2023, 1, 17, 5, 31, 41, 149167), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:31:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:31:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:31:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:41 UTC)" executed successfully builder stdout | 2023-01-17 05:31:41,212 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:41,224 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:41,224 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:41,235 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:31:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:31:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:06.941185+00:00 (in 23.999188 seconds) namespacegcworker stdout | 2023-01-17 05:31:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:42 UTC)" (scheduled at 2023-01-17 05:31:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:31:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:31:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:32:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:55.646354+00:00 (in 9.999425 seconds) notificationworker stdout | 2023-01-17 05:31:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:55 UTC)" (scheduled at 2023-01-17 05:31:45.646354+00:00) notificationworker stdout | 2023-01-17 05:31:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 45, 647396), True, datetime.datetime(2023, 1, 17, 5, 31, 45, 647396), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:31:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:31:49.537472+00:00 (in 2.600019 seconds) securityworker stdout | 2023-01-17 05:31:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:16 UTC)" (scheduled at 2023-01-17 05:31:46.936988+00:00) securityworker stdout | 2023-01-17 05:31:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:46,954 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:46,957 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:46,957 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:31:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:31:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:16.936988+00:00 (in 27.398964 seconds) securityworker stdout | 2023-01-17 05:31:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:19 UTC)" (scheduled at 2023-01-17 05:31:49.537472+00:00) securityworker stdout | 2023-01-17 05:31:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:31:49,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:49,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:31:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:31:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:31:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:31:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:31:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:31:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:31:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:31:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:31:49,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:31:49,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:31:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:31:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:19 UTC)" executed successfully builder stdout | 2023-01-17 05:31:51,245 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:31:51,257 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:31:51,257 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:31:51,267 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:31:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:31:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:21.348605+00:00 (in 29.999453 seconds) gcworker stdout | 2023-01-17 05:31:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:21 UTC)" (scheduled at 2023-01-17 05:31:51.348605+00:00) gcworker stdout | 2023-01-17 05:31:51,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:31:51,361 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723911360, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:31:51,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:31:51,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:31:51,366 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:31:52,059 [231] [DEBUG] [app] Starting request: urn:request:1f599b8e-eb5a-43e3-acd9-e47ac9af82eb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:52,060 [236] [DEBUG] [app] Starting request: urn:request:cde9f977-c6a9-4e15-96f4-87110522cf5f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:31:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:31:52,064 [242] [DEBUG] [app] Starting request: urn:request:323bc4b6-f016-4b91-bafc-25b3b987e9b9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:52,064 [230] [DEBUG] [app] Starting request: urn:request:a6ac24b6-e529-440d-8831-bbf5cd5edae3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:31:52,065 [242] [DEBUG] [app] Ending request: urn:request:323bc4b6-f016-4b91-bafc-25b3b987e9b9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:323bc4b6-f016-4b91-bafc-25b3b987e9b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:52,065 [230] [DEBUG] [app] Ending request: urn:request:a6ac24b6-e529-440d-8831-bbf5cd5edae3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a6ac24b6-e529-440d-8831-bbf5cd5edae3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:31:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:31:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:31:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:52,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:31:52,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:31:52,070 [233] [DEBUG] [app] Starting request: urn:request:e5cff81a-3671-45fe-bc40-06972dbed165 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:52,070 [231] [DEBUG] [app] Starting request: urn:request:1fb8d1e9-8c32-405c-a697-24dfc44d1cc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:31:52,070 [233] [DEBUG] [app] Ending request: urn:request:e5cff81a-3671-45fe-bc40-06972dbed165 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5cff81a-3671-45fe-bc40-06972dbed165', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:52,070 [231] [DEBUG] [app] Ending request: urn:request:1fb8d1e9-8c32-405c-a697-24dfc44d1cc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1fb8d1e9-8c32-405c-a697-24dfc44d1cc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:31:52,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:31:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:31:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:31:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:31:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:31:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:31:52,087 [231] [DEBUG] [app] Ending request: urn:request:1f599b8e-eb5a-43e3-acd9-e47ac9af82eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1f599b8e-eb5a-43e3-acd9-e47ac9af82eb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:52,087 [236] [DEBUG] [app] Ending request: urn:request:cde9f977-c6a9-4e15-96f4-87110522cf5f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cde9f977-c6a9-4e15-96f4-87110522cf5f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:31:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:31:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:31:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:31:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 05:31:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:31:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:28.963662+00:00 (in 33.999749 seconds) exportactionlogsworker stdout | 2023-01-17 05:31:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:54 UTC)" (scheduled at 2023-01-17 05:31:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:31:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 31, 54, 964341), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:31:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 54, 964341), True, datetime.datetime(2023, 1, 17, 5, 31, 54, 964341), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:31:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 31, 54, 964341), True, datetime.datetime(2023, 1, 17, 5, 31, 54, 964341), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 31, 54, 964341), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:31:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:31:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:36:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:31:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:01.636362+00:00 (in 5.989486 seconds) notificationworker stdout | 2023-01-17 05:31:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:05 UTC)" (scheduled at 2023-01-17 05:31:55.646354+00:00) notificationworker stdout | 2023-01-17 05:31:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:31:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 31, 55, 647381), True, datetime.datetime(2023, 1, 17, 5, 31, 55, 647381), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:31:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:31:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:31:55,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:31:58,818 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:31:58,957 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:31:59,846 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:32:00,287 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:32:00,342 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:32:00,654 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:32:00,804 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:32:01,066 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:32:01,068 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:32:01,071 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:32:01,074 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:32:01,076 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:32:01,277 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:01,290 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:01,290 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:01,301 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:05.646354+00:00 (in 4.009454 seconds) notificationworker stdout | 2023-01-17 05:32:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:01 UTC)" (scheduled at 2023-01-17 05:32:01.636362+00:00) notificationworker stdout | 2023-01-17 05:32:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:32:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:32:03,990 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:32:04,185 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:32:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:32:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:19.346120+00:00 (in 14.408841 seconds) repositorygcworker stdout | 2023-01-17 05:32:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:04 UTC)" (scheduled at 2023-01-17 05:32:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:32:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:32:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 4, 937805), True, datetime.datetime(2023, 1, 17, 5, 32, 4, 937805), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:32:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:32:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:32:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:32:05,053 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:32:05,056 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:32:05,058 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:32:05,254 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:32:05,287 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:32:05,378 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:32:05,380 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:32:05,383 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:32:05,385 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:32:05,390 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:32:05,393 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:32:05,396 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:32:05,398 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:32:05,402 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:32:05,499 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:32:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:15.646354+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 05:32:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:15 UTC)" (scheduled at 2023-01-17 05:32:05.646354+00:00) notificationworker stdout | 2023-01-17 05:32:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 5, 647197), True, datetime.datetime(2023, 1, 17, 5, 32, 5, 647197), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:32:06,094 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:32:06,250 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:32:06,449 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:32:06,550 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} namespacegcworker stdout | 2023-01-17 05:32:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:32:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:41.148245+00:00 (in 34.206588 seconds) namespacegcworker stdout | 2023-01-17 05:32:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:37:06 UTC)" (scheduled at 2023-01-17 05:32:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:32:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 32, 6, 942055), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:32:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 32, 6, 942055), True, datetime.datetime(2023, 1, 17, 5, 32, 6, 942055), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:32:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 32, 6, 942055), True, datetime.datetime(2023, 1, 17, 5, 32, 6, 942055), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 32, 6, 942055), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:32:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:32:06,961 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:37:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:32:07,060 [233] [DEBUG] [app] Starting request: urn:request:493aaf47-ff35-4a32-9480-ab8304e7cbcf (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:07,060 [231] [DEBUG] [app] Starting request: urn:request:edcfa7c7-6c78-46a3-b2be-6ed412582a2d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:07,065 [241] [DEBUG] [app] Starting request: urn:request:2f4e5d35-4113-4acc-9eb2-0c91eba8bfb5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,066 [241] [DEBUG] [app] Ending request: urn:request:2f4e5d35-4113-4acc-9eb2-0c91eba8bfb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f4e5d35-4113-4acc-9eb2-0c91eba8bfb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,066 [230] [DEBUG] [app] Starting request: urn:request:f82a31be-5c8d-4bb3-b3dc-4ac561aacc98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:32:07,066 [230] [DEBUG] [app] Ending request: urn:request:f82a31be-5c8d-4bb3-b3dc-4ac561aacc98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f82a31be-5c8d-4bb3-b3dc-4ac561aacc98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:07,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:32:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:07,070 [231] [DEBUG] [app] Starting request: urn:request:cbd520a2-cad5-42c3-8030-ffca2605da38 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:07,070 [231] [DEBUG] [app] Ending request: urn:request:cbd520a2-cad5-42c3-8030-ffca2605da38 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cbd520a2-cad5-42c3-8030-ffca2605da38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:07,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:07,072 [236] [DEBUG] [app] Starting request: urn:request:8d993f24-b020-4385-926d-d5424139fa7e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:07,072 [236] [DEBUG] [app] Ending request: urn:request:8d993f24-b020-4385-926d-d5424139fa7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d993f24-b020-4385-926d-d5424139fa7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:07,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:07,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:07,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:07,087 [233] [DEBUG] [app] Ending request: urn:request:493aaf47-ff35-4a32-9480-ab8304e7cbcf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:493aaf47-ff35-4a32-9480-ab8304e7cbcf', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:07,087 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:32:07,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:32:07,088 [231] [DEBUG] [app] Ending request: urn:request:edcfa7c7-6c78-46a3-b2be-6ed412582a2d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:edcfa7c7-6c78-46a3-b2be-6ed412582a2d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:07,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:32:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" globalpromstats stdout | 2023-01-17 05:32:07,160 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:32:07,696 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:32:07,714 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:32:07,981 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:32:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:32:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:40.738568+00:00 (in 29.999572 seconds) buildlogsarchiver stdout | 2023-01-17 05:32:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:40 UTC)" (scheduled at 2023-01-17 05:32:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:32:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 32, 10, 739408), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:32:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:32:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:32:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:40 UTC)" executed successfully builder stdout | 2023-01-17 05:32:11,307 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:11,319 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:11,319 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:11,330 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:25.646354+00:00 (in 9.999536 seconds) notificationworker stdout | 2023-01-17 05:32:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:25 UTC)" (scheduled at 2023-01-17 05:32:15.646354+00:00) notificationworker stdout | 2023-01-17 05:32:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 15, 647243), True, datetime.datetime(2023, 1, 17, 5, 32, 15, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:19.537472+00:00 (in 2.599944 seconds) securityworker stdout | 2023-01-17 05:32:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:46 UTC)" (scheduled at 2023-01-17 05:32:16.936988+00:00) securityworker stdout | 2023-01-17 05:32:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:32:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:32:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:04.936835+00:00 (in 45.590169 seconds) repositorygcworker stdout | 2023-01-17 05:32:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:19 UTC)" (scheduled at 2023-01-17 05:32:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:32:19,347 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:32:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:46.936988+00:00 (in 27.399047 seconds) securityworker stdout | 2023-01-17 05:32:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:49 UTC)" (scheduled at 2023-01-17 05:32:19.537472+00:00) securityworker stdout | 2023-01-17 05:32:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:32:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:32:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:19,559 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:19,559 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:32:19,559 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:32:19,560 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:32:19,560 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:19,561 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:19,561 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:19,561 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:32:19,561 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:49 UTC)" executed successfully builder stdout | 2023-01-17 05:32:21,334 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:21,347 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:21,347 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gcworker stdout | 2023-01-17 05:32:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:32:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:51.348605+00:00 (in 29.999531 seconds) gcworker stdout | 2023-01-17 05:32:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:51 UTC)" (scheduled at 2023-01-17 05:32:21.348605+00:00) gcworker stdout | 2023-01-17 05:32:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723941349, 1209600, True, 3, 500, 1, 0]) builder stdout | 2023-01-17 05:32:21,357 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:32:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:32:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:32:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:32:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:32:22,060 [235] [DEBUG] [app] Starting request: urn:request:b2487327-b401-4ce8-b331-919f82c627f1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:22,060 [233] [DEBUG] [app] Starting request: urn:request:90021392-9eda-428e-800a-aca4a3c1f4e3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:22,065 [243] [DEBUG] [app] Starting request: urn:request:d719e19e-ea7d-4dff-804e-7ec15d879f21 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:22,065 [230] [DEBUG] [app] Starting request: urn:request:8528cef9-2351-4b0e-b248-20ee523244f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:22,066 [243] [DEBUG] [app] Ending request: urn:request:d719e19e-ea7d-4dff-804e-7ec15d879f21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d719e19e-ea7d-4dff-804e-7ec15d879f21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:22,066 [230] [DEBUG] [app] Ending request: urn:request:8528cef9-2351-4b0e-b248-20ee523244f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8528cef9-2351-4b0e-b248-20ee523244f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:32:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:32:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:22,071 [231] [DEBUG] [app] Starting request: urn:request:b36d122a-d489-437a-98e9-8cdc26fff4df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:22,071 [231] [DEBUG] [app] Ending request: urn:request:b36d122a-d489-437a-98e9-8cdc26fff4df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b36d122a-d489-437a-98e9-8cdc26fff4df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:22,071 [236] [DEBUG] [app] Starting request: urn:request:885d9efe-f9fd-46b4-ba37-cde774f34216 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:22,072 [236] [DEBUG] [app] Ending request: urn:request:885d9efe-f9fd-46b4-ba37-cde774f34216 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:885d9efe-f9fd-46b4-ba37-cde774f34216', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:32:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:22,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:22,088 [235] [DEBUG] [app] Ending request: urn:request:b2487327-b401-4ce8-b331-919f82c627f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2487327-b401-4ce8-b331-919f82c627f1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:22,088 [233] [DEBUG] [app] Ending request: urn:request:90021392-9eda-428e-800a-aca4a3c1f4e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90021392-9eda-428e-800a-aca4a3c1f4e3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:22,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:32:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:32:22,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:32:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:35.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 05:32:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:35 UTC)" (scheduled at 2023-01-17 05:32:25.646354+00:00) notificationworker stdout | 2023-01-17 05:32:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 25, 647178), True, datetime.datetime(2023, 1, 17, 5, 32, 25, 647178), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:32:28,852 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} exportactionlogsworker stdout | 2023-01-17 05:32:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:32:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:35.853396+00:00 (in 6.889269 seconds) exportactionlogsworker stdout | 2023-01-17 05:32:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:28 UTC)" (scheduled at 2023-01-17 05:32:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:32:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:32:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:28 UTC)" executed successfully gcworker stdout | 2023-01-17 05:32:28,993 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:32:29,871 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:32:30,322 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:32:30,376 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:32:30,671 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:32:30,835 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:32:31,075 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:32:31,078 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:32:31,080 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:32:31,083 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:32:31,085 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:32:31,368 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:31,380 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:31,381 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:31,391 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:32:34,026 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:32:34,206 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:32:35,061 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:32:35,064 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:32:35,066 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:32:35,286 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:32:35,307 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:32:35,388 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:32:35,391 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:32:35,393 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:32:35,396 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:32:35,398 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:32:35,401 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:32:35,405 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:32:35,408 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:32:35,410 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:32:35,534 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:32:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:45.646354+00:00 (in 9.999537 seconds) notificationworker stdout | 2023-01-17 05:32:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:45 UTC)" (scheduled at 2023-01-17 05:32:35.646354+00:00) notificationworker stdout | 2023-01-17 05:32:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 35, 647258), True, datetime.datetime(2023, 1, 17, 5, 32, 35, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:32:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:32:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:28.963662+00:00 (in 53.109758 seconds) exportactionlogsworker stdout | 2023-01-17 05:32:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:35 UTC)" (scheduled at 2023-01-17 05:32:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:32:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:32:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 35, 854365), True, datetime.datetime(2023, 1, 17, 5, 32, 35, 854365), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:32:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:32:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:32:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:32:36,102 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:32:36,258 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:32:36,484 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:32:36,584 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:32:37,060 [236] [DEBUG] [app] Starting request: urn:request:0ab4d1c3-906f-4e6a-b774-2e2ff82d2529 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:37,060 [231] [DEBUG] [app] Starting request: urn:request:a2ef2e73-db39-465c-a6d4-55869b1b2fce (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:37,066 [242] [DEBUG] [app] Starting request: urn:request:fab37e2b-a7ff-4654-8fe6-5b903d592817 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,066 [241] [DEBUG] [app] Starting request: urn:request:4c700a11-f45d-46fe-9cef-815791106ed7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,066 [242] [DEBUG] [app] Ending request: urn:request:fab37e2b-a7ff-4654-8fe6-5b903d592817 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fab37e2b-a7ff-4654-8fe6-5b903d592817', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,066 [241] [DEBUG] [app] Ending request: urn:request:4c700a11-f45d-46fe-9cef-815791106ed7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c700a11-f45d-46fe-9cef-815791106ed7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:32:37,067 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:32:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:37,071 [235] [DEBUG] [app] Starting request: urn:request:871f3677-94ab-41e8-aee5-5fb776cc70d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:37,072 [233] [DEBUG] [app] Starting request: urn:request:16825ecb-1e7c-442d-9bf6-947200633702 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:37,072 [235] [DEBUG] [app] Ending request: urn:request:871f3677-94ab-41e8-aee5-5fb776cc70d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:871f3677-94ab-41e8-aee5-5fb776cc70d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:32:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:37,072 [233] [DEBUG] [app] Ending request: urn:request:16825ecb-1e7c-442d-9bf6-947200633702 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16825ecb-1e7c-442d-9bf6-947200633702', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:37,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:37,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:37,089 [236] [DEBUG] [app] Ending request: urn:request:0ab4d1c3-906f-4e6a-b774-2e2ff82d2529 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ab4d1c3-906f-4e6a-b774-2e2ff82d2529', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:37,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:37,089 [231] [DEBUG] [app] Ending request: urn:request:a2ef2e73-db39-465c-a6d4-55869b1b2fce (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2ef2e73-db39-465c-a6d4-55869b1b2fce', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:37,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:32:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:32:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) globalpromstats stdout | 2023-01-17 05:32:37,195 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:32:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:32:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:37.537407+00:00 (in 0.280402 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:32:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:37 UTC)" (scheduled at 2023-01-17 05:32:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:32:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:32:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 37, 257385), True, datetime.datetime(2023, 1, 17, 5, 32, 37, 257385), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:32:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:32:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:32:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:32:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:32:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:37.256652+00:00 (in 59.718896 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:32:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:37 UTC)" (scheduled at 2023-01-17 05:32:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:32:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:32:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:32:37,719 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:32:37,721 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:32:37,992 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:32:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:32:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:10.738568+00:00 (in 29.999566 seconds) buildlogsarchiver stdout | 2023-01-17 05:32:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:10 UTC)" (scheduled at 2023-01-17 05:32:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:32:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 32, 40, 739411), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:32:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:32:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:32:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:32:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:32:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:42.941536+00:00 (in 1.792789 seconds) namespacegcworker stdout | 2023-01-17 05:32:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:41 UTC)" (scheduled at 2023-01-17 05:32:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:32:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:32:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 41, 149227), True, datetime.datetime(2023, 1, 17, 5, 32, 41, 149227), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:32:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:32:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:32:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:41 UTC)" executed successfully builder stdout | 2023-01-17 05:32:41,401 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:41,414 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:41,414 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:41,425 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:32:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:32:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:41.148245+00:00 (in 58.206218 seconds) namespacegcworker stdout | 2023-01-17 05:32:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:42 UTC)" (scheduled at 2023-01-17 05:32:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:32:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:32:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:33:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:32:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:55.646354+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 05:32:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:55 UTC)" (scheduled at 2023-01-17 05:32:45.646354+00:00) notificationworker stdout | 2023-01-17 05:32:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 45, 647213), True, datetime.datetime(2023, 1, 17, 5, 32, 45, 647213), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:32:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:32:49.537472+00:00 (in 2.600034 seconds) securityworker stdout | 2023-01-17 05:32:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:16 UTC)" (scheduled at 2023-01-17 05:32:46.936988+00:00) securityworker stdout | 2023-01-17 05:32:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:32:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:32:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:16.936988+00:00 (in 27.399044 seconds) securityworker stdout | 2023-01-17 05:32:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:19 UTC)" (scheduled at 2023-01-17 05:32:49.537472+00:00) securityworker stdout | 2023-01-17 05:32:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:32:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:32:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:32:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:32:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:32:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:32:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:32:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:32:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:32:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:32:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:32:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:32:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:32:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:32:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:32:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:21.348605+00:00 (in 29.999536 seconds) gcworker stdout | 2023-01-17 05:32:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:21 UTC)" (scheduled at 2023-01-17 05:32:51.348605+00:00) gcworker stdout | 2023-01-17 05:32:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672723971349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:32:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:32:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:32:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:21 UTC)" executed successfully builder stdout | 2023-01-17 05:32:51,435 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:32:51,447 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:32:51,447 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:32:51,458 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:52,060 [235] [DEBUG] [app] Starting request: urn:request:35f7cb3e-063d-4744-bbb2-a5d3a773ebca (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:52,060 [231] [DEBUG] [app] Starting request: urn:request:87fdc276-e132-4776-a78b-c9080776facc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:32:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:32:52,065 [242] [DEBUG] [app] Starting request: urn:request:0415c712-20c2-4d60-91d0-146caa4c73ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:52,065 [241] [DEBUG] [app] Starting request: urn:request:a566a0ee-9863-4797-954b-c759372ad0c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:32:52,065 [242] [DEBUG] [app] Ending request: urn:request:0415c712-20c2-4d60-91d0-146caa4c73ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0415c712-20c2-4d60-91d0-146caa4c73ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:32:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:32:52,066 [241] [DEBUG] [app] Ending request: urn:request:a566a0ee-9863-4797-954b-c759372ad0c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a566a0ee-9863-4797-954b-c759372ad0c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:32:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:32:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:32:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:32:52,070 [236] [DEBUG] [app] Starting request: urn:request:00b9fd84-ffc4-4b9a-be35-89de846c4471 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:52,070 [231] [DEBUG] [app] Starting request: urn:request:f3091b74-47f5-480a-b74e-272b4f9d9a6f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:32:52,070 [236] [DEBUG] [app] Ending request: urn:request:00b9fd84-ffc4-4b9a-be35-89de846c4471 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00b9fd84-ffc4-4b9a-be35-89de846c4471', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:52,070 [231] [DEBUG] [app] Ending request: urn:request:f3091b74-47f5-480a-b74e-272b4f9d9a6f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3091b74-47f5-480a-b74e-272b4f9d9a6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:32:52,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:52,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:32:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:32:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:32:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:32:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:32:52,087 [231] [DEBUG] [app] Ending request: urn:request:87fdc276-e132-4776-a78b-c9080776facc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:87fdc276-e132-4776-a78b-c9080776facc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:52,087 [235] [DEBUG] [app] Ending request: urn:request:35f7cb3e-063d-4744-bbb2-a5d3a773ebca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:35f7cb3e-063d-4744-bbb2-a5d3a773ebca', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:32:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:32:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:32:52,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:32:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:32:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:32:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:01.636362+00:00 (in 5.989562 seconds) notificationworker stdout | 2023-01-17 05:32:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:05 UTC)" (scheduled at 2023-01-17 05:32:55.646354+00:00) notificationworker stdout | 2023-01-17 05:32:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:32:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 32, 55, 647228), True, datetime.datetime(2023, 1, 17, 5, 32, 55, 647228), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:32:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:32:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:32:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:32:58,870 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:32:59,028 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:32:59,907 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:33:00,355 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:33:00,411 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:33:00,705 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:33:00,870 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:33:01,082 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:33:01,086 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:33:01,088 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:33:01,090 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:33:01,093 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:33:01,468 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:01,481 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:01,481 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:01,491 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:05.646354+00:00 (in 4.009535 seconds) notificationworker stdout | 2023-01-17 05:33:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:01 UTC)" (scheduled at 2023-01-17 05:33:01.636362+00:00) notificationworker stdout | 2023-01-17 05:33:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:33:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:01 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:33:04,037 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:33:04,227 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:33:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:33:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:19.346120+00:00 (in 14.408795 seconds) repositorygcworker stdout | 2023-01-17 05:33:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:04 UTC)" (scheduled at 2023-01-17 05:33:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:33:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:33:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 4, 937776), True, datetime.datetime(2023, 1, 17, 5, 33, 4, 937776), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:33:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:33:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:33:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:33:05,068 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:33:05,071 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:33:05,073 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:33:05,309 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:33:05,342 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:33:05,399 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:33:05,402 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:33:05,404 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:33:05,406 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:33:05,408 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:33:05,411 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:33:05,414 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:33:05,417 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:33:05,419 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:33:05,569 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:33:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:15.646354+00:00 (in 9.999583 seconds) notificationworker stdout | 2023-01-17 05:33:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:15 UTC)" (scheduled at 2023-01-17 05:33:05.646354+00:00) notificationworker stdout | 2023-01-17 05:33:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 5, 647205), True, datetime.datetime(2023, 1, 17, 5, 33, 5, 647205), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:33:06,119 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:33:06,275 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:33:06,506 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:33:06,590 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:33:07,060 [236] [DEBUG] [app] Starting request: urn:request:cb2277fd-51cb-4475-8db6-0d5e00196137 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:07,060 [233] [DEBUG] [app] Starting request: urn:request:1747cbc4-78aa-40f2-9aa8-b00ffde1ee75 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:07,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:07,065 [230] [DEBUG] [app] Starting request: urn:request:9c5cc2b5-ace5-42fa-94ba-811633c3be32 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:07,066 [241] [DEBUG] [app] Starting request: urn:request:d6b2bf23-a3d4-49c0-92f6-7d6634176b29 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:07,066 [230] [DEBUG] [app] Ending request: urn:request:9c5cc2b5-ace5-42fa-94ba-811633c3be32 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c5cc2b5-ace5-42fa-94ba-811633c3be32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:33:07,066 [241] [DEBUG] [app] Ending request: urn:request:d6b2bf23-a3d4-49c0-92f6-7d6634176b29 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6b2bf23-a3d4-49c0-92f6-7d6634176b29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:33:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:33:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:07,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:07,070 [231] [DEBUG] [app] Starting request: urn:request:93b3b858-15f6-4e33-8e43-6a9f10352394 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:07,071 [231] [DEBUG] [app] Ending request: urn:request:93b3b858-15f6-4e33-8e43-6a9f10352394 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93b3b858-15f6-4e33-8e43-6a9f10352394', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:07,071 [235] [DEBUG] [app] Starting request: urn:request:a8783b4b-a0e1-4270-ae68-46b5d214c4c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:33:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:07,071 [235] [DEBUG] [app] Ending request: urn:request:a8783b4b-a0e1-4270-ae68-46b5d214c4c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a8783b4b-a0e1-4270-ae68-46b5d214c4c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:07,089 [233] [DEBUG] [app] Ending request: urn:request:1747cbc4-78aa-40f2-9aa8-b00ffde1ee75 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1747cbc4-78aa-40f2-9aa8-b00ffde1ee75', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:07,089 [236] [DEBUG] [app] Ending request: urn:request:cb2277fd-51cb-4475-8db6-0d5e00196137 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb2277fd-51cb-4475-8db6-0d5e00196137', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:07,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:07,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:07,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:33:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) globalpromstats stdout | 2023-01-17 05:33:07,201 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:33:07,742 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:33:07,751 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:33:08,026 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:33:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:33:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:40.738568+00:00 (in 29.999560 seconds) buildlogsarchiver stdout | 2023-01-17 05:33:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:40 UTC)" (scheduled at 2023-01-17 05:33:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:33:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 33, 10, 739365), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:33:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:33:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:33:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:40 UTC)" executed successfully builder stdout | 2023-01-17 05:33:11,492 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:11,504 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:11,504 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:11,515 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:25.646354+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 05:33:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:25 UTC)" (scheduled at 2023-01-17 05:33:15.646354+00:00) notificationworker stdout | 2023-01-17 05:33:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 15, 647199), True, datetime.datetime(2023, 1, 17, 5, 33, 15, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:15,669 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:15,669 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:15,669 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:19.537472+00:00 (in 2.600012 seconds) securityworker stdout | 2023-01-17 05:33:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:46 UTC)" (scheduled at 2023-01-17 05:33:16.936988+00:00) securityworker stdout | 2023-01-17 05:33:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:33:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:33:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:04.936835+00:00 (in 45.590230 seconds) repositorygcworker stdout | 2023-01-17 05:33:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:19 UTC)" (scheduled at 2023-01-17 05:33:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:33:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:33:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:46.936988+00:00 (in 27.399044 seconds) securityworker stdout | 2023-01-17 05:33:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:49 UTC)" (scheduled at 2023-01-17 05:33:19.537472+00:00) securityworker stdout | 2023-01-17 05:33:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:33:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:33:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:19,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:19,556 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:19,556 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:33:19,556 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:33:19,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:33:19,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:33:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:33:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:33:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:51.348605+00:00 (in 29.999510 seconds) gcworker stdout | 2023-01-17 05:33:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:51 UTC)" (scheduled at 2023-01-17 05:33:21.348605+00:00) gcworker stdout | 2023-01-17 05:33:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724001349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:33:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:33:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:33:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:33:51 UTC)" executed successfully builder stdout | 2023-01-17 05:33:21,525 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:21,538 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:21,538 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:21,548 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:22,060 [231] [DEBUG] [app] Starting request: urn:request:9f42ce10-0979-4be4-b7ee-2515f305cce3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:22,060 [236] [DEBUG] [app] Starting request: urn:request:414dd085-0cb6-41f9-a088-199f864adbf5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:22,065 [242] [DEBUG] [app] Starting request: urn:request:6408f4e8-c544-4bf6-b8d4-6c21ebab4e9e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:22,065 [241] [DEBUG] [app] Starting request: urn:request:ba2dfcbe-8af8-43c0-ac20-a7803853b7b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:22,065 [242] [DEBUG] [app] Ending request: urn:request:6408f4e8-c544-4bf6-b8d4-6c21ebab4e9e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6408f4e8-c544-4bf6-b8d4-6c21ebab4e9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:33:22,066 [241] [DEBUG] [app] Ending request: urn:request:ba2dfcbe-8af8-43c0-ac20-a7803853b7b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ba2dfcbe-8af8-43c0-ac20-a7803853b7b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:33:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:33:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:22,070 [235] [DEBUG] [app] Starting request: urn:request:d74941a2-7e81-49a3-a926-bdc575519e30 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:22,070 [235] [DEBUG] [app] Ending request: urn:request:d74941a2-7e81-49a3-a926-bdc575519e30 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d74941a2-7e81-49a3-a926-bdc575519e30', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:22,070 [233] [DEBUG] [app] Starting request: urn:request:2327b55d-c1f2-4b4f-94a1-2088f4d0c5c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:22,071 [233] [DEBUG] [app] Ending request: urn:request:2327b55d-c1f2-4b4f-94a1-2088f4d0c5c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2327b55d-c1f2-4b4f-94a1-2088f4d0c5c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:33:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:22,087 [236] [DEBUG] [app] Ending request: urn:request:414dd085-0cb6-41f9-a088-199f864adbf5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:414dd085-0cb6-41f9-a088-199f864adbf5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:22,087 [231] [DEBUG] [app] Ending request: urn:request:9f42ce10-0979-4be4-b7ee-2515f305cce3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f42ce10-0979-4be4-b7ee-2515f305cce3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:22,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:33:22,087 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:33:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) notificationworker stdout | 2023-01-17 05:33:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:35.646354+00:00 (in 9.999568 seconds) notificationworker stdout | 2023-01-17 05:33:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:35 UTC)" (scheduled at 2023-01-17 05:33:25.646354+00:00) notificationworker stdout | 2023-01-17 05:33:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 25, 647217), True, datetime.datetime(2023, 1, 17, 5, 33, 25, 647217), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:28,906 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} exportactionlogsworker stdout | 2023-01-17 05:33:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:33:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:35.853396+00:00 (in 6.889236 seconds) exportactionlogsworker stdout | 2023-01-17 05:33:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:28 UTC)" (scheduled at 2023-01-17 05:33:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:33:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:33:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:28 UTC)" executed successfully gcworker stdout | 2023-01-17 05:33:29,058 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:33:29,943 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:33:30,391 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:33:30,440 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:33:30,737 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:33:30,902 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:33:31,088 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:33:31,091 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:33:31,094 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:33:31,097 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:33:31,099 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:33:31,558 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:31,571 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:31,571 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:31,581 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:33:34,064 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:33:34,239 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:33:35,075 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:33:35,077 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:33:35,079 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:33:35,318 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:33:35,362 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:33:35,408 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:33:35,410 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:33:35,413 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:33:35,415 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:33:35,417 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:33:35,420 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:33:35,423 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:33:35,425 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:33:35,427 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:33:35,602 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:33:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:36.635997+00:00 (in 0.989191 seconds) notificationworker stdout | 2023-01-17 05:33:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:45 UTC)" (scheduled at 2023-01-17 05:33:35.646354+00:00) notificationworker stdout | 2023-01-17 05:33:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 35, 647261), True, datetime.datetime(2023, 1, 17, 5, 33, 35, 647261), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:33:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:33:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:28.963662+00:00 (in 53.109734 seconds) exportactionlogsworker stdout | 2023-01-17 05:33:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:35 UTC)" (scheduled at 2023-01-17 05:33:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:33:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:33:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 35, 854366), True, datetime.datetime(2023, 1, 17, 5, 33, 35, 854366), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:33:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:33:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:33:35,866 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:33:36,155 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:33:36,310 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:33:36,534 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:33:36,598 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} notificationworker stdout | 2023-01-17 05:33:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:45.646354+00:00 (in 9.009925 seconds) notificationworker stdout | 2023-01-17 05:33:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:38:36 UTC)" (scheduled at 2023-01-17 05:33:36.635997+00:00) notificationworker stdout | 2023-01-17 05:33:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 33, 36, 636844), 'notification/%']) notificationworker stdout | 2023-01-17 05:33:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 33, 36, 636844), True, datetime.datetime(2023, 1, 17, 5, 33, 36, 636844), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:33:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 33, 36, 636844), True, datetime.datetime(2023, 1, 17, 5, 33, 36, 636844), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 33, 36, 636844), 'notification/%']) notificationworker stdout | 2023-01-17 05:33:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:38:36 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:33:37,059 [236] [DEBUG] [app] Starting request: urn:request:f1e224a9-db28-4941-940b-2cb07a405089 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:37,059 [231] [DEBUG] [app] Starting request: urn:request:40b48f3c-d9a6-492f-9813-36af649d9447 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:37,060 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:37,064 [242] [DEBUG] [app] Starting request: urn:request:2d2c5e92-d078-4fb8-b79b-05941cdd167a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:37,065 [242] [DEBUG] [app] Ending request: urn:request:2d2c5e92-d078-4fb8-b79b-05941cdd167a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d2c5e92-d078-4fb8-b79b-05941cdd167a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:37,065 [241] [DEBUG] [app] Starting request: urn:request:4a856060-9514-42ed-abf8-ba5e897e8a7a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:33:37,065 [241] [DEBUG] [app] Ending request: urn:request:4a856060-9514-42ed-abf8-ba5e897e8a7a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a856060-9514-42ed-abf8-ba5e897e8a7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:33:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:33:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:37,070 [233] [DEBUG] [app] Starting request: urn:request:9bc7a55d-7f3a-4b81-911b-ae07978a0c9c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:37,070 [233] [DEBUG] [app] Ending request: urn:request:9bc7a55d-7f3a-4b81-911b-ae07978a0c9c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9bc7a55d-7f3a-4b81-911b-ae07978a0c9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:37,070 [235] [DEBUG] [app] Starting request: urn:request:44a39767-562b-40d0-bb51-87cfd2f9fe72 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:37,071 [235] [DEBUG] [app] Ending request: urn:request:44a39767-562b-40d0-bb51-87cfd2f9fe72 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:44a39767-562b-40d0-bb51-87cfd2f9fe72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:33:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:37,087 [231] [DEBUG] [app] Ending request: urn:request:40b48f3c-d9a6-492f-9813-36af649d9447 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:40b48f3c-d9a6-492f-9813-36af649d9447', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:33:37,087 [236] [DEBUG] [app] Ending request: urn:request:f1e224a9-db28-4941-940b-2cb07a405089 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1e224a9-db28-4941-940b-2cb07a405089', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) globalpromstats stdout | 2023-01-17 05:33:37,216 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:33:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:33:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:37.537407+00:00 (in 0.280363 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:33:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:37 UTC)" (scheduled at 2023-01-17 05:33:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:33:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:33:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 37, 257493), True, datetime.datetime(2023, 1, 17, 5, 33, 37, 257493), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:33:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:33:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:33:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:33:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:33:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:37.256652+00:00 (in 59.718811 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:33:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:37 UTC)" (scheduled at 2023-01-17 05:33:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:33:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:33:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:33:37,761 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:33:37,786 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:33:38,062 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:33:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:33:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:10.738568+00:00 (in 29.999592 seconds) buildlogsarchiver stdout | 2023-01-17 05:33:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:10 UTC)" (scheduled at 2023-01-17 05:33:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:33:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 33, 40, 739357), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:33:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:33:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:33:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:33:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:33:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:42.941536+00:00 (in 1.792791 seconds) namespacegcworker stdout | 2023-01-17 05:33:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:41 UTC)" (scheduled at 2023-01-17 05:33:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:33:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:33:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 41, 149200), True, datetime.datetime(2023, 1, 17, 5, 33, 41, 149200), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:33:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:33:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:33:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:41 UTC)" executed successfully builder stdout | 2023-01-17 05:33:41,584 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:41,596 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:41,596 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:41,607 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:33:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:33:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:41.148245+00:00 (in 58.206170 seconds) namespacegcworker stdout | 2023-01-17 05:33:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:42 UTC)" (scheduled at 2023-01-17 05:33:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:33:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:33:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:34:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:55.646354+00:00 (in 9.999523 seconds) notificationworker stdout | 2023-01-17 05:33:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:55 UTC)" (scheduled at 2023-01-17 05:33:45.646354+00:00) notificationworker stdout | 2023-01-17 05:33:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 45, 647273), True, datetime.datetime(2023, 1, 17, 5, 33, 45, 647273), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:33:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:33:49.537472+00:00 (in 2.600015 seconds) securityworker stdout | 2023-01-17 05:33:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:16 UTC)" (scheduled at 2023-01-17 05:33:46.936988+00:00) securityworker stdout | 2023-01-17 05:33:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:33:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:33:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:16.936988+00:00 (in 27.399036 seconds) securityworker stdout | 2023-01-17 05:33:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:19 UTC)" (scheduled at 2023-01-17 05:33:49.537472+00:00) securityworker stdout | 2023-01-17 05:33:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:33:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:33:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:33:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:33:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:33:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:33:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:33:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:33:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:33:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:33:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:33:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:33:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:33:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:33:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:33:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:21.348605+00:00 (in 29.999500 seconds) gcworker stdout | 2023-01-17 05:33:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:21 UTC)" (scheduled at 2023-01-17 05:33:51.348605+00:00) gcworker stdout | 2023-01-17 05:33:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724031349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:33:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:33:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:33:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:21 UTC)" executed successfully builder stdout | 2023-01-17 05:33:51,617 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:33:51,630 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:33:51,630 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:33:51,641 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:52,060 [231] [DEBUG] [app] Starting request: urn:request:47407c19-87f6-4263-b555-b238a8ba6aee (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:52,060 [236] [DEBUG] [app] Starting request: urn:request:276e2c7f-7058-435f-b0c8-7eca5bc443a4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:33:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:33:52,065 [230] [DEBUG] [app] Starting request: urn:request:73c1e8d7-04d2-4159-bff6-318cb01d8cd3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,065 [243] [DEBUG] [app] Starting request: urn:request:1fbda99c-177c-4cf2-915d-33c2d5c5188d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,066 [230] [DEBUG] [app] Ending request: urn:request:73c1e8d7-04d2-4159-bff6-318cb01d8cd3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:73c1e8d7-04d2-4159-bff6-318cb01d8cd3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,066 [243] [DEBUG] [app] Ending request: urn:request:1fbda99c-177c-4cf2-915d-33c2d5c5188d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1fbda99c-177c-4cf2-915d-33c2d5c5188d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:33:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:33:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:33:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:33:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:33:52,070 [233] [DEBUG] [app] Starting request: urn:request:4586ebf3-b739-434a-a5bc-1aba697bd58f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:52,071 [235] [DEBUG] [app] Starting request: urn:request:34fa87a2-9d82-4033-a206-9d7c151fa8e6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:33:52,071 [233] [DEBUG] [app] Ending request: urn:request:4586ebf3-b739-434a-a5bc-1aba697bd58f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4586ebf3-b739-434a-a5bc-1aba697bd58f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:33:52,071 [235] [DEBUG] [app] Ending request: urn:request:34fa87a2-9d82-4033-a206-9d7c151fa8e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:34fa87a2-9d82-4033-a206-9d7c151fa8e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:33:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:33:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:33:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:33:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:33:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:33:52,087 [236] [DEBUG] [app] Ending request: urn:request:276e2c7f-7058-435f-b0c8-7eca5bc443a4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:276e2c7f-7058-435f-b0c8-7eca5bc443a4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:52,087 [231] [DEBUG] [app] Ending request: urn:request:47407c19-87f6-4263-b555-b238a8ba6aee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47407c19-87f6-4263-b555-b238a8ba6aee', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:33:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:33:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:33:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:33:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) notificationworker stdout | 2023-01-17 05:33:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:33:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:01.636362+00:00 (in 5.989563 seconds) notificationworker stdout | 2023-01-17 05:33:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:05 UTC)" (scheduled at 2023-01-17 05:33:55.646354+00:00) notificationworker stdout | 2023-01-17 05:33:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:33:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 33, 55, 647249), True, datetime.datetime(2023, 1, 17, 5, 33, 55, 647249), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:33:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:33:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:33:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:33:58,924 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:33:59,066 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:33:59,951 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:34:00,413 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:34:00,456 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:34:00,772 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:34:00,925 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:34:01,096 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:34:01,099 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:34:01,102 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:34:01,105 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:34:01,107 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:34:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:05.646354+00:00 (in 4.009519 seconds) notificationworker stdout | 2023-01-17 05:34:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:01 UTC)" (scheduled at 2023-01-17 05:34:01.636362+00:00) notificationworker stdout | 2023-01-17 05:34:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:34:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:01 UTC)" executed successfully builder stdout | 2023-01-17 05:34:01,650 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:01,663 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:01,663 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:01,674 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:34:04,090 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:34:04,254 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:34:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:34:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:19.346120+00:00 (in 14.408787 seconds) repositorygcworker stdout | 2023-01-17 05:34:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:04 UTC)" (scheduled at 2023-01-17 05:34:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:34:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:34:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 4, 937804), True, datetime.datetime(2023, 1, 17, 5, 34, 4, 937804), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:34:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:34:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:34:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:34:05,082 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:34:05,084 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:34:05,087 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:34:05,339 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:34:05,397 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:34:05,417 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:34:05,419 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:34:05,422 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:34:05,424 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:34:05,427 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:34:05,429 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:34:05,433 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:34:05,435 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:34:05,437 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} repositorygcworker stdout | 2023-01-17 05:34:05,637 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:34:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:15.646354+00:00 (in 9.999479 seconds) notificationworker stdout | 2023-01-17 05:34:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:15 UTC)" (scheduled at 2023-01-17 05:34:05.646354+00:00) notificationworker stdout | 2023-01-17 05:34:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 5, 647227), True, datetime.datetime(2023, 1, 17, 5, 34, 5, 647227), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:15 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:34:06,190 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:34:06,345 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:34:06,565 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:34:06,632 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:34:07,060 [231] [DEBUG] [app] Starting request: urn:request:d00edebb-5b68-4222-9828-d640f0f1726f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:07,060 [235] [DEBUG] [app] Starting request: urn:request:109e36c7-23fb-41a1-abfd-f9b27a9de37b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:07,064 [243] [DEBUG] [app] Starting request: urn:request:0e667174-4804-4255-9bef-345de046fa91 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:07,064 [230] [DEBUG] [app] Starting request: urn:request:5a0902b4-6075-4638-819b-4756f4cfa75b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:07,065 [243] [DEBUG] [app] Ending request: urn:request:0e667174-4804-4255-9bef-345de046fa91 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e667174-4804-4255-9bef-345de046fa91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:07,065 [230] [DEBUG] [app] Ending request: urn:request:5a0902b4-6075-4638-819b-4756f4cfa75b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a0902b4-6075-4638-819b-4756f4cfa75b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:34:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:07,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:07,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:07,069 [236] [DEBUG] [app] Starting request: urn:request:b568c61b-aed9-49ae-8e6c-95f6df32c82c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:07,069 [233] [DEBUG] [app] Starting request: urn:request:1fd28cbb-4d15-457e-9258-abb8c0e975f8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:07,070 [236] [DEBUG] [app] Ending request: urn:request:b568c61b-aed9-49ae-8e6c-95f6df32c82c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b568c61b-aed9-49ae-8e6c-95f6df32c82c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:07,070 [233] [DEBUG] [app] Ending request: urn:request:1fd28cbb-4d15-457e-9258-abb8c0e975f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1fd28cbb-4d15-457e-9258-abb8c0e975f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:34:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:07,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:07,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:07,087 [235] [DEBUG] [app] Ending request: urn:request:109e36c7-23fb-41a1-abfd-f9b27a9de37b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:109e36c7-23fb-41a1-abfd-f9b27a9de37b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:07,087 [231] [DEBUG] [app] Ending request: urn:request:d00edebb-5b68-4222-9828-d640f0f1726f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d00edebb-5b68-4222-9828-d640f0f1726f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:34:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) globalpromstats stdout | 2023-01-17 05:34:07,225 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} blobuploadcleanupworker stdout | 2023-01-17 05:34:07,762 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run blobuploadcleanupworker stdout | 2023-01-17 05:34:07,763 [59] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2023-01-17 05:34:07 UTC)" (scheduled at 2023-01-17 05:34:07.762380+00:00) blobuploadcleanupworker stdout | 2023-01-17 05:34:07,763 [59] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:34:07,763 [59] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ... blobuploadcleanupworker stdout | 2023-01-17 05:34:07,763 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:34:07.762380+00:00 (in 3599.999678 seconds) blobuploadcleanupworker stdout | 2023-01-17 05:34:07,765 [59] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2023-01-17 05:34:07,765 [59] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:34:07,765 [59] [DEBUG] [__main__] Performing blob upload cleanup blobuploadcleanupworker stdout | 2023-01-17 05:34:07,766 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 15, 5, 34, 7, 765731), 1, 0]) blobuploadcleanupworker stdout | 2023-01-17 05:34:07,776 [59] [DEBUG] [__main__] No additional stale blob uploads found blobuploadcleanupworker stdout | 2023-01-17 05:34:07,776 [59] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 05:34:07,776 [59] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:34:07,776 [59] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'. blobuploadcleanupworker stdout | 2023-01-17 05:34:07,777 [59] [DEBUG] [util.locking] Released lock BLOB_CLEANUP blobuploadcleanupworker stdout | 2023-01-17 05:34:07,778 [59] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2023-01-17 06:34:07 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:34:07,778 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:34:07,799 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:34:08,093 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:34:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:34:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:40.738568+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 05:34:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:40 UTC)" (scheduled at 2023-01-17 05:34:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:34:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 34, 10, 739466), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:34:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:34:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:34:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:40 UTC)" executed successfully builder stdout | 2023-01-17 05:34:11,684 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:11,697 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:11,697 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:11,708 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:25.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:34:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:25 UTC)" (scheduled at 2023-01-17 05:34:15.646354+00:00) notificationworker stdout | 2023-01-17 05:34:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 15, 647280), True, datetime.datetime(2023, 1, 17, 5, 34, 15, 647280), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:19.537472+00:00 (in 2.600000 seconds) securityworker stdout | 2023-01-17 05:34:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:46 UTC)" (scheduled at 2023-01-17 05:34:16.936988+00:00) securityworker stdout | 2023-01-17 05:34:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:34:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:34:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:04.936835+00:00 (in 45.590273 seconds) repositorygcworker stdout | 2023-01-17 05:34:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:19 UTC)" (scheduled at 2023-01-17 05:34:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:34:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:34:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:46.936988+00:00 (in 27.399042 seconds) securityworker stdout | 2023-01-17 05:34:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:49 UTC)" (scheduled at 2023-01-17 05:34:19.537472+00:00) securityworker stdout | 2023-01-17 05:34:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:34:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:34:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:34:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:34:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:34:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:34:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:34:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:34:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:51.348605+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 05:34:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:51 UTC)" (scheduled at 2023-01-17 05:34:21.348605+00:00) gcworker stdout | 2023-01-17 05:34:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724061349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:34:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:34:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:34:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:34:51 UTC)" executed successfully builder stdout | 2023-01-17 05:34:21,718 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:21,731 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:21,731 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:21,742 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:22,060 [231] [DEBUG] [app] Starting request: urn:request:024518ec-83db-41c5-917e-2fb0a1e8d331 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:22,060 [236] [DEBUG] [app] Starting request: urn:request:a207a8d8-9519-407b-a72c-c1ccdf701df4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:22,065 [230] [DEBUG] [app] Starting request: urn:request:d558a88c-979a-4381-b00a-9521e22920cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:22,065 [243] [DEBUG] [app] Starting request: urn:request:44cd24b3-392f-4b58-be6c-c141be4a3c75 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:22,066 [230] [DEBUG] [app] Ending request: urn:request:d558a88c-979a-4381-b00a-9521e22920cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d558a88c-979a-4381-b00a-9521e22920cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:22,066 [243] [DEBUG] [app] Ending request: urn:request:44cd24b3-392f-4b58-be6c-c141be4a3c75 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44cd24b3-392f-4b58-be6c-c141be4a3c75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:34:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:34:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:22,070 [233] [DEBUG] [app] Starting request: urn:request:7a2702ec-fec6-4bd4-8956-95118eeb6a97 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:22,070 [235] [DEBUG] [app] Starting request: urn:request:3aee57f1-3f78-4dbe-9e2c-350a08d88cc3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:22,070 [233] [DEBUG] [app] Ending request: urn:request:7a2702ec-fec6-4bd4-8956-95118eeb6a97 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a2702ec-fec6-4bd4-8956-95118eeb6a97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:22,071 [235] [DEBUG] [app] Ending request: urn:request:3aee57f1-3f78-4dbe-9e2c-350a08d88cc3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3aee57f1-3f78-4dbe-9e2c-350a08d88cc3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:34:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:22,087 [231] [DEBUG] [app] Ending request: urn:request:024518ec-83db-41c5-917e-2fb0a1e8d331 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:024518ec-83db-41c5-917e-2fb0a1e8d331', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:22,087 [236] [DEBUG] [app] Ending request: urn:request:a207a8d8-9519-407b-a72c-c1ccdf701df4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a207a8d8-9519-407b-a72c-c1ccdf701df4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:34:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) notificationworker stdout | 2023-01-17 05:34:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:35.646354+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 05:34:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:35 UTC)" (scheduled at 2023-01-17 05:34:25.646354+00:00) notificationworker stdout | 2023-01-17 05:34:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 25, 647220), True, datetime.datetime(2023, 1, 17, 5, 34, 25, 647220), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:35 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:28,960 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} exportactionlogsworker stdout | 2023-01-17 05:34:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:34:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:35.853396+00:00 (in 6.889322 seconds) exportactionlogsworker stdout | 2023-01-17 05:34:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:28 UTC)" (scheduled at 2023-01-17 05:34:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:34:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:34:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:28 UTC)" executed successfully gcworker stdout | 2023-01-17 05:34:29,090 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:34:29,960 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:34:30,449 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:34:30,461 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:34:30,806 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:34:30,959 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:34:31,102 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:34:31,105 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:34:31,108 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:34:31,111 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:34:31,113 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:34:31,742 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:31,755 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:31,756 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:31,767 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:34:34,126 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:34:34,262 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:34:35,090 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:34:35,092 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:34:35,094 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:34:35,375 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:34:35,425 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:34:35,427 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} servicekey stdout | 2023-01-17 05:34:35,430 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-registry stdout | 2023-01-17 05:34:35,434 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:34:35,436 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:34:35,439 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:34:35,441 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:34:35,444 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:34:35,446 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:34:35,448 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:34:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:45.646354+00:00 (in 9.999559 seconds) notificationworker stdout | 2023-01-17 05:34:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:45 UTC)" (scheduled at 2023-01-17 05:34:35.646354+00:00) notificationworker stdout | 2023-01-17 05:34:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 35, 647188), True, datetime.datetime(2023, 1, 17, 5, 34, 35, 647188), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:45 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:34:35,667 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} exportactionlogsworker stdout | 2023-01-17 05:34:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:34:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:28.963662+00:00 (in 53.109830 seconds) exportactionlogsworker stdout | 2023-01-17 05:34:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:35 UTC)" (scheduled at 2023-01-17 05:34:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:34:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:34:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 35, 854214), True, datetime.datetime(2023, 1, 17, 5, 34, 35, 854214), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:34:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:34:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:34:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:34:36,216 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:34:36,380 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:34:36,600 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:34:36,667 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:34:37,060 [231] [DEBUG] [app] Starting request: urn:request:2e16547f-a874-4504-b4f5-710c10736f87 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:37,060 [236] [DEBUG] [app] Starting request: urn:request:a677b9b3-c399-4672-9f20-7f58d238ce9a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:37,065 [230] [DEBUG] [app] Starting request: urn:request:1ea67905-92ba-40a0-990c-ff679c293fad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:37,066 [243] [DEBUG] [app] Starting request: urn:request:51255703-bf2c-4405-8681-0d71b45a2030 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:37,066 [230] [DEBUG] [app] Ending request: urn:request:1ea67905-92ba-40a0-990c-ff679c293fad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1ea67905-92ba-40a0-990c-ff679c293fad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:37,066 [243] [DEBUG] [app] Ending request: urn:request:51255703-bf2c-4405-8681-0d71b45a2030 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51255703-bf2c-4405-8681-0d71b45a2030', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:34:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:34:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:37,070 [235] [DEBUG] [app] Starting request: urn:request:a6c73883-58e4-46d9-8935-ede27ba76d68 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:37,070 [233] [DEBUG] [app] Starting request: urn:request:63a529a0-3af5-4413-8e6e-e502f0f6c92f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:37,071 [235] [DEBUG] [app] Ending request: urn:request:a6c73883-58e4-46d9-8935-ede27ba76d68 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6c73883-58e4-46d9-8935-ede27ba76d68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:37,071 [233] [DEBUG] [app] Ending request: urn:request:63a529a0-3af5-4413-8e6e-e502f0f6c92f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:63a529a0-3af5-4413-8e6e-e502f0f6c92f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:34:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:37,088 [231] [DEBUG] [app] Ending request: urn:request:2e16547f-a874-4504-b4f5-710c10736f87 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e16547f-a874-4504-b4f5-710c10736f87', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:37,088 [236] [DEBUG] [app] Ending request: urn:request:a677b9b3-c399-4672-9f20-7f58d238ce9a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a677b9b3-c399-4672-9f20-7f58d238ce9a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:34:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:34:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" globalpromstats stdout | 2023-01-17 05:34:37,231 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:34:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:34:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:37.537407+00:00 (in 0.280219 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:34:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:37 UTC)" (scheduled at 2023-01-17 05:34:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:34:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:34:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 37, 257567), True, datetime.datetime(2023, 1, 17, 5, 34, 37, 257567), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:34:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:34:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:34:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:34:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:34:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:37.256652+00:00 (in 59.718888 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:34:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:37 UTC)" (scheduled at 2023-01-17 05:34:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:34:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:34:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:34:37,814 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:34:37,816 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:34:38,102 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:34:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:34:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:10.738568+00:00 (in 29.999536 seconds) buildlogsarchiver stdout | 2023-01-17 05:34:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:10 UTC)" (scheduled at 2023-01-17 05:34:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:34:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 34, 40, 739388), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:34:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:34:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:34:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:34:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:34:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:42.941536+00:00 (in 1.792742 seconds) namespacegcworker stdout | 2023-01-17 05:34:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:41 UTC)" (scheduled at 2023-01-17 05:34:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:34:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:34:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 41, 149203), True, datetime.datetime(2023, 1, 17, 5, 34, 41, 149203), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:34:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:34:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:34:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:41 UTC)" executed successfully builder stdout | 2023-01-17 05:34:41,777 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:41,790 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:41,790 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:41,801 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:34:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:34:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:41.148245+00:00 (in 58.206251 seconds) namespacegcworker stdout | 2023-01-17 05:34:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:42 UTC)" (scheduled at 2023-01-17 05:34:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:34:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:34:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:35:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:55.646354+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 05:34:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:55 UTC)" (scheduled at 2023-01-17 05:34:45.646354+00:00) notificationworker stdout | 2023-01-17 05:34:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 45, 647210), True, datetime.datetime(2023, 1, 17, 5, 34, 45, 647210), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:34:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:34:49.537472+00:00 (in 2.600002 seconds) securityworker stdout | 2023-01-17 05:34:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:16 UTC)" (scheduled at 2023-01-17 05:34:46.936988+00:00) securityworker stdout | 2023-01-17 05:34:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:34:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:34:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:16.936988+00:00 (in 27.399004 seconds) securityworker stdout | 2023-01-17 05:34:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:19 UTC)" (scheduled at 2023-01-17 05:34:49.537472+00:00) securityworker stdout | 2023-01-17 05:34:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:34:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:34:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:34:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:34:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:34:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:34:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:34:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:34:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:34:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:34:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:34:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:34:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:34:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:34:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:34:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:21.348605+00:00 (in 29.999526 seconds) gcworker stdout | 2023-01-17 05:34:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:21 UTC)" (scheduled at 2023-01-17 05:34:51.348605+00:00) gcworker stdout | 2023-01-17 05:34:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724091349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:34:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:34:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:34:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:21 UTC)" executed successfully builder stdout | 2023-01-17 05:34:51,811 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:34:51,824 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:34:51,824 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:34:51,834 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:52,060 [235] [DEBUG] [app] Starting request: urn:request:1c43d5fa-744e-418b-86bc-799f1a1a8a15 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:52,060 [236] [DEBUG] [app] Starting request: urn:request:8967f3e2-1089-4caa-9139-6582df6a597a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:34:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:52,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:34:52,066 [230] [DEBUG] [app] Starting request: urn:request:59f8e48c-2b8f-49c8-a4ff-a5b043bb576b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,066 [243] [DEBUG] [app] Starting request: urn:request:3cead244-807e-43c8-94fd-bb45ec1128a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,066 [230] [DEBUG] [app] Ending request: urn:request:59f8e48c-2b8f-49c8-a4ff-a5b043bb576b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59f8e48c-2b8f-49c8-a4ff-a5b043bb576b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,066 [243] [DEBUG] [app] Ending request: urn:request:3cead244-807e-43c8-94fd-bb45ec1128a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3cead244-807e-43c8-94fd-bb45ec1128a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:34:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:34:52,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:34:52,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:34:52,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:34:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:34:52,071 [231] [DEBUG] [app] Starting request: urn:request:13208910-da53-4acb-b28c-180de8f3b237 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:52,071 [233] [DEBUG] [app] Starting request: urn:request:eaf5ffaa-10e3-467a-a1df-9569051e7bf2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:34:52,071 [231] [DEBUG] [app] Ending request: urn:request:13208910-da53-4acb-b28c-180de8f3b237 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:13208910-da53-4acb-b28c-180de8f3b237', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:34:52,071 [233] [DEBUG] [app] Ending request: urn:request:eaf5ffaa-10e3-467a-a1df-9569051e7bf2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eaf5ffaa-10e3-467a-a1df-9569051e7bf2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:34:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:34:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:34:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:34:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:34:52,088 [236] [DEBUG] [app] Ending request: urn:request:8967f3e2-1089-4caa-9139-6582df6a597a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8967f3e2-1089-4caa-9139-6582df6a597a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:34:52,096 [235] [DEBUG] [app] Ending request: urn:request:1c43d5fa-744e-418b-86bc-799f1a1a8a15 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c43d5fa-744e-418b-86bc-799f1a1a8a15', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:34:52,096 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:34:52,096 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:34:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.037) notificationworker stdout | 2023-01-17 05:34:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:34:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:01.636362+00:00 (in 5.989528 seconds) notificationworker stdout | 2023-01-17 05:34:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:05 UTC)" (scheduled at 2023-01-17 05:34:55.646354+00:00) notificationworker stdout | 2023-01-17 05:34:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:34:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 34, 55, 647224), True, datetime.datetime(2023, 1, 17, 5, 34, 55, 647224), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:34:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:34:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:34:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:34:58,996 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:34:59,125 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:34:59,995 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:35:00,469 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:35:00,472 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:35:00,840 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:35:00,994 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:35:01,109 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:35:01,112 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:35:01,114 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:35:01,117 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:35:01,119 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:35:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:05.646354+00:00 (in 4.009516 seconds) notificationworker stdout | 2023-01-17 05:35:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:01 UTC)" (scheduled at 2023-01-17 05:35:01.636362+00:00) notificationworker stdout | 2023-01-17 05:35:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:35:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:01 UTC)" executed successfully builder stdout | 2023-01-17 05:35:01,835 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:01,848 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:01,848 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:01,859 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:35:04,155 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:35:04,297 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:35:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:35:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:19.346120+00:00 (in 14.408820 seconds) repositorygcworker stdout | 2023-01-17 05:35:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:04 UTC)" (scheduled at 2023-01-17 05:35:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:35:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:35:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 4, 937760), True, datetime.datetime(2023, 1, 17, 5, 35, 4, 937760), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:35:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:35:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:35:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:35:05,097 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:35:05,099 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:35:05,102 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:35:05,403 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:35:05,431 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:35:05,434 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:35:05,441 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:35:05,443 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:35:05,445 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:35:05,450 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:35:05,452 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:35:05,454 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:35:05,457 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} servicekey stdout | 2023-01-17 05:35:05,464 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:35:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:15.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:35:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:15 UTC)" (scheduled at 2023-01-17 05:35:05.646354+00:00) notificationworker stdout | 2023-01-17 05:35:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 5, 647205), True, datetime.datetime(2023, 1, 17, 5, 35, 5, 647205), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:15 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:35:05,686 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:35:06,251 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:35:06,388 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:35:06,622 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:35:06,672 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:35:07,059 [231] [DEBUG] [app] Starting request: urn:request:238e65c3-fe14-49ec-8dba-82bdefb97957 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:07,060 [233] [DEBUG] [app] Starting request: urn:request:0cde2fd4-c132-4fcc-aad6-e9eb132a0bfe (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:07,065 [230] [DEBUG] [app] Starting request: urn:request:50575af2-54d0-4d03-89c1-0c2a51eccd44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:07,065 [242] [DEBUG] [app] Starting request: urn:request:330e0619-6532-44a5-88aa-dd83deb359f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:07,065 [230] [DEBUG] [app] Ending request: urn:request:50575af2-54d0-4d03-89c1-0c2a51eccd44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50575af2-54d0-4d03-89c1-0c2a51eccd44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:35:07,066 [242] [DEBUG] [app] Ending request: urn:request:330e0619-6532-44a5-88aa-dd83deb359f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:330e0619-6532-44a5-88aa-dd83deb359f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:35:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:35:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:07,070 [236] [DEBUG] [app] Starting request: urn:request:c1ea0cd7-41df-4180-85e1-4d57e3b0f0b4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:07,070 [231] [DEBUG] [app] Starting request: urn:request:c35101b8-d66b-4ce9-b85c-9a687b1872f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:07,070 [236] [DEBUG] [app] Ending request: urn:request:c1ea0cd7-41df-4180-85e1-4d57e3b0f0b4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c1ea0cd7-41df-4180-85e1-4d57e3b0f0b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:07,071 [231] [DEBUG] [app] Ending request: urn:request:c35101b8-d66b-4ce9-b85c-9a687b1872f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c35101b8-d66b-4ce9-b85c-9a687b1872f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:07,088 [231] [DEBUG] [app] Ending request: urn:request:238e65c3-fe14-49ec-8dba-82bdefb97957 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:238e65c3-fe14-49ec-8dba-82bdefb97957', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:07,088 [233] [DEBUG] [app] Ending request: urn:request:0cde2fd4-c132-4fcc-aad6-e9eb132a0bfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0cde2fd4-c132-4fcc-aad6-e9eb132a0bfe', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:35:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) globalpromstats stdout | 2023-01-17 05:35:07,238 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:35:07,822 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:35:07,848 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:35:08,138 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:35:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:35:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:40.738568+00:00 (in 29.999548 seconds) buildlogsarchiver stdout | 2023-01-17 05:35:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:40 UTC)" (scheduled at 2023-01-17 05:35:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:35:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 35, 10, 739415), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:35:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:35:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:35:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:40 UTC)" executed successfully builder stdout | 2023-01-17 05:35:11,869 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:11,881 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:11,881 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:11,892 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:25.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:35:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:25 UTC)" (scheduled at 2023-01-17 05:35:15.646354+00:00) notificationworker stdout | 2023-01-17 05:35:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 15, 647219), True, datetime.datetime(2023, 1, 17, 5, 35, 15, 647219), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:35:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:19.537472+00:00 (in 2.600013 seconds) securityworker stdout | 2023-01-17 05:35:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:46 UTC)" (scheduled at 2023-01-17 05:35:16.936988+00:00) securityworker stdout | 2023-01-17 05:35:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:35:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:35:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:04.936835+00:00 (in 45.590225 seconds) repositorygcworker stdout | 2023-01-17 05:35:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:19 UTC)" (scheduled at 2023-01-17 05:35:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:35:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:35:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:35:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:46.936988+00:00 (in 27.399013 seconds) securityworker stdout | 2023-01-17 05:35:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:49 UTC)" (scheduled at 2023-01-17 05:35:19.537472+00:00) securityworker stdout | 2023-01-17 05:35:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:35:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:35:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:35:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:35:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:35:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:35:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:35:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:35:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:51.348605+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 05:35:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:51 UTC)" (scheduled at 2023-01-17 05:35:21.348605+00:00) gcworker stdout | 2023-01-17 05:35:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724121349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:35:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:35:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:35:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:35:51 UTC)" executed successfully builder stdout | 2023-01-17 05:35:21,902 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:21,914 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:21,914 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:21,925 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:22,061 [236] [DEBUG] [app] Starting request: urn:request:7c997303-ba21-498b-9f90-b96ebcad6a98 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:22,061 [235] [DEBUG] [app] Starting request: urn:request:c9064ef0-d11c-4820-9f11-a38c5a167be8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:22,066 [242] [DEBUG] [app] Starting request: urn:request:a913b246-6f24-4827-bac8-645f7ca98906 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:22,066 [241] [DEBUG] [app] Starting request: urn:request:85f82550-336b-481d-ac3c-1d4d15c56dff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:22,066 [242] [DEBUG] [app] Ending request: urn:request:a913b246-6f24-4827-bac8-645f7ca98906 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a913b246-6f24-4827-bac8-645f7ca98906', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:22,067 [241] [DEBUG] [app] Ending request: urn:request:85f82550-336b-481d-ac3c-1d4d15c56dff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85f82550-336b-481d-ac3c-1d4d15c56dff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:22,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:35:22,067 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:35:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:22,071 [231] [DEBUG] [app] Starting request: urn:request:3ddcb78c-2779-41b8-9ca6-c5890b34e5fe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:22,071 [233] [DEBUG] [app] Starting request: urn:request:b2fa9eb3-b654-4c44-9eca-f4d3dfe8a372 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:22,071 [231] [DEBUG] [app] Ending request: urn:request:3ddcb78c-2779-41b8-9ca6-c5890b34e5fe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ddcb78c-2779-41b8-9ca6-c5890b34e5fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:22,071 [233] [DEBUG] [app] Ending request: urn:request:b2fa9eb3-b654-4c44-9eca-f4d3dfe8a372 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2fa9eb3-b654-4c44-9eca-f4d3dfe8a372', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:22,087 [236] [DEBUG] [app] Ending request: urn:request:7c997303-ba21-498b-9f90-b96ebcad6a98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c997303-ba21-498b-9f90-b96ebcad6a98', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:22,088 [235] [DEBUG] [app] Ending request: urn:request:c9064ef0-d11c-4820-9f11-a38c5a167be8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9064ef0-d11c-4820-9f11-a38c5a167be8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:35:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:35:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:35.646354+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 05:35:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:35 UTC)" (scheduled at 2023-01-17 05:35:25.646354+00:00) notificationworker stdout | 2023-01-17 05:35:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 25, 647272), True, datetime.datetime(2023, 1, 17, 5, 35, 25, 647272), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:35:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:35:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:35.853396+00:00 (in 6.889243 seconds) exportactionlogsworker stdout | 2023-01-17 05:35:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:28 UTC)" (scheduled at 2023-01-17 05:35:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:35:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:35:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:29,031 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:35:29,156 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:35:30,019 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:35:30,506 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:35:30,508 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:35:30,871 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:35:31,029 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:35:31,115 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:35:31,119 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:35:31,123 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:35:31,125 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:35:31,128 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:35:31,929 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:31,942 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:31,942 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:31,952 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:35:34,183 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:35:34,317 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:35:35,103 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:35:35,106 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:35:35,108 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} storagereplication stdout | 2023-01-17 05:35:35,424 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:35:35,437 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:35:35,440 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:35:35,447 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:35:35,450 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:35:35,455 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:35:35,457 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:35:35,460 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:35:35,463 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:35:35,467 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} servicekey stdout | 2023-01-17 05:35:35,499 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:35:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:45.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 05:35:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:45 UTC)" (scheduled at 2023-01-17 05:35:35.646354+00:00) notificationworker stdout | 2023-01-17 05:35:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 35, 647169), True, datetime.datetime(2023, 1, 17, 5, 35, 35, 647169), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:45 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:35:35,700 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} exportactionlogsworker stdout | 2023-01-17 05:35:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:35:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:28.963662+00:00 (in 53.109774 seconds) exportactionlogsworker stdout | 2023-01-17 05:35:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:35 UTC)" (scheduled at 2023-01-17 05:35:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:35:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:35:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 35, 854287), True, datetime.datetime(2023, 1, 17, 5, 35, 35, 854287), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:35:35,866 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:35:35,866 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:35:35,867 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:35:36,279 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:35:36,406 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:35:36,656 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:35:36,694 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:35:37,060 [233] [DEBUG] [app] Starting request: urn:request:0d61ff6e-2c8d-4916-bfd9-58f16a467ad9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:37,060 [235] [DEBUG] [app] Starting request: urn:request:d7668913-6cad-443f-87d7-93919e78d2ae (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:37,064 [230] [DEBUG] [app] Starting request: urn:request:5d60b59e-4fb0-49f1-9609-bb24d688479d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:37,065 [230] [DEBUG] [app] Ending request: urn:request:5d60b59e-4fb0-49f1-9609-bb24d688479d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d60b59e-4fb0-49f1-9609-bb24d688479d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:37,065 [243] [DEBUG] [app] Starting request: urn:request:bc1292a9-4e07-4139-939e-3231a75e4edf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:35:37,066 [243] [DEBUG] [app] Ending request: urn:request:bc1292a9-4e07-4139-939e-3231a75e4edf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bc1292a9-4e07-4139-939e-3231a75e4edf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:37,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:35:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:35:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:37,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:37,069 [236] [DEBUG] [app] Starting request: urn:request:5410f976-fe00-46c5-a2aa-78d1cad7b478 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:37,070 [236] [DEBUG] [app] Ending request: urn:request:5410f976-fe00-46c5-a2aa-78d1cad7b478 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5410f976-fe00-46c5-a2aa-78d1cad7b478', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:37,070 [231] [DEBUG] [app] Starting request: urn:request:9d132678-a5c8-48d0-98f0-94b1e8aed11f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:37,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:35:37,070 [231] [DEBUG] [app] Ending request: urn:request:9d132678-a5c8-48d0-98f0-94b1e8aed11f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9d132678-a5c8-48d0-98f0-94b1e8aed11f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:37,087 [233] [DEBUG] [app] Ending request: urn:request:0d61ff6e-2c8d-4916-bfd9-58f16a467ad9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d61ff6e-2c8d-4916-bfd9-58f16a467ad9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:37,087 [235] [DEBUG] [app] Ending request: urn:request:d7668913-6cad-443f-87d7-93919e78d2ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d7668913-6cad-443f-87d7-93919e78d2ae', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:37,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:37,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:35:37,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:35:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:35:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:37.537407+00:00 (in 0.280387 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:35:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:37 UTC)" (scheduled at 2023-01-17 05:35:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:35:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:35:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 37, 257425), True, datetime.datetime(2023, 1, 17, 5, 35, 37, 257425), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:35:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:35:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:35:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:35:37,272 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:35:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:35:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:23.537048+00:00 (in 45.999193 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:35:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:37 UTC)" (scheduled at 2023-01-17 05:35:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:35:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:35:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:35:37,857 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:35:37,859 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:35:38,150 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:35:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:35:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:10.738568+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 05:35:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:10 UTC)" (scheduled at 2023-01-17 05:35:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:35:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 35, 40, 739472), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:35:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:35:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:35:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:35:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:35:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:42.941536+00:00 (in 1.792786 seconds) namespacegcworker stdout | 2023-01-17 05:35:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:41 UTC)" (scheduled at 2023-01-17 05:35:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:35:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:35:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 41, 149202), True, datetime.datetime(2023, 1, 17, 5, 35, 41, 149202), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:35:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:35:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:35:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:41 UTC)" executed successfully builder stdout | 2023-01-17 05:35:41,962 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:41,975 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:41,975 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:41,986 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:35:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:35:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:41.148245+00:00 (in 58.206203 seconds) namespacegcworker stdout | 2023-01-17 05:35:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:42 UTC)" (scheduled at 2023-01-17 05:35:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:35:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:35:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:36:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:55.646354+00:00 (in 9.999558 seconds) notificationworker stdout | 2023-01-17 05:35:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:55 UTC)" (scheduled at 2023-01-17 05:35:45.646354+00:00) notificationworker stdout | 2023-01-17 05:35:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 45, 647232), True, datetime.datetime(2023, 1, 17, 5, 35, 45, 647232), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:35:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:35:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:35:49.537472+00:00 (in 2.600009 seconds) securityworker stdout | 2023-01-17 05:35:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:16 UTC)" (scheduled at 2023-01-17 05:35:46.936988+00:00) securityworker stdout | 2023-01-17 05:35:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:35:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:35:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:16.936988+00:00 (in 27.399063 seconds) securityworker stdout | 2023-01-17 05:35:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:19 UTC)" (scheduled at 2023-01-17 05:35:49.537472+00:00) securityworker stdout | 2023-01-17 05:35:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:35:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:35:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:35:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:35:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:35:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:35:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:35:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:35:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:35:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:35:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:35:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:35:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:35:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:35:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:35:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:21.348605+00:00 (in 29.999493 seconds) gcworker stdout | 2023-01-17 05:35:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:21 UTC)" (scheduled at 2023-01-17 05:35:51.348605+00:00) gcworker stdout | 2023-01-17 05:35:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724151349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:35:51,369 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:35:51,369 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:35:51,369 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:21 UTC)" executed successfully builder stdout | 2023-01-17 05:35:51,996 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:35:52,008 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:35:52,008 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:35:52,019 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:52,060 [233] [DEBUG] [app] Starting request: urn:request:ba18a73f-1c55-4aa5-9287-18f89155496d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:52,060 [231] [DEBUG] [app] Starting request: urn:request:90f3b9d7-ccb7-470c-b434-0606914014fe (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:35:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:35:52,065 [230] [DEBUG] [app] Starting request: urn:request:b9c27067-93a1-4d02-9bf5-b1990ab170e5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:52,065 [243] [DEBUG] [app] Starting request: urn:request:8cd08988-148b-41eb-bbc7-68af4e248c64 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:35:52,066 [230] [DEBUG] [app] Ending request: urn:request:b9c27067-93a1-4d02-9bf5-b1990ab170e5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9c27067-93a1-4d02-9bf5-b1990ab170e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:35:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:35:52,066 [243] [DEBUG] [app] Ending request: urn:request:8cd08988-148b-41eb-bbc7-68af4e248c64 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8cd08988-148b-41eb-bbc7-68af4e248c64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:35:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:35:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:35:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:35:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:35:52,070 [236] [DEBUG] [app] Starting request: urn:request:4f969f05-6005-4d3b-8700-32b0bf523bb8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:52,071 [236] [DEBUG] [app] Ending request: urn:request:4f969f05-6005-4d3b-8700-32b0bf523bb8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f969f05-6005-4d3b-8700-32b0bf523bb8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:35:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:52,071 [235] [DEBUG] [app] Starting request: urn:request:0a5eae64-9463-4e2d-977a-cf8f1f551664 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:35:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:52,071 [235] [DEBUG] [app] Ending request: urn:request:0a5eae64-9463-4e2d-977a-cf8f1f551664 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a5eae64-9463-4e2d-977a-cf8f1f551664', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:35:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:35:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:35:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:35:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:35:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:35:52,093 [233] [DEBUG] [app] Ending request: urn:request:ba18a73f-1c55-4aa5-9287-18f89155496d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba18a73f-1c55-4aa5-9287-18f89155496d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:52,093 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:52,093 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.034 47 0.034) gunicorn-web stdout | 2023-01-17 05:35:52,095 [231] [DEBUG] [app] Ending request: urn:request:90f3b9d7-ccb7-470c-b434-0606914014fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90f3b9d7-ccb7-470c-b434-0606914014fe', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:35:52,095 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:35:52,095 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:35:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) notificationworker stdout | 2023-01-17 05:35:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:35:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:01.636362+00:00 (in 5.989514 seconds) notificationworker stdout | 2023-01-17 05:35:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:05 UTC)" (scheduled at 2023-01-17 05:35:55.646354+00:00) notificationworker stdout | 2023-01-17 05:35:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:35:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 35, 55, 647242), True, datetime.datetime(2023, 1, 17, 5, 35, 55, 647242), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:35:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:35:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:35:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:35:59,067 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:35:59,191 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:36:00,026 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:36:00,535 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:36:00,538 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:36:00,880 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:36:01,063 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:36:01,122 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:36:01,125 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:36:01,129 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:36:01,133 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:36:01,135 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:36:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:05.646354+00:00 (in 4.009516 seconds) notificationworker stdout | 2023-01-17 05:36:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:01 UTC)" (scheduled at 2023-01-17 05:36:01.636362+00:00) notificationworker stdout | 2023-01-17 05:36:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:36:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:01 UTC)" executed successfully builder stdout | 2023-01-17 05:36:02,029 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:02,042 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:02,042 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:02,052 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:36:04,219 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:36:04,323 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:36:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:36:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:17.345601+00:00 (in 12.408304 seconds) repositorygcworker stdout | 2023-01-17 05:36:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:04 UTC)" (scheduled at 2023-01-17 05:36:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:36:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:36:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 4, 937806), True, datetime.datetime(2023, 1, 17, 5, 36, 4, 937806), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:36:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:36:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:36:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:36:05,111 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:36:05,114 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:36:05,116 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:36:05,444 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:36:05,447 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} storagereplication stdout | 2023-01-17 05:36:05,453 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-registry stdout | 2023-01-17 05:36:05,455 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:36:05,459 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:36:05,464 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:36:05,468 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:36:05,470 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:36:05,473 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:36:05,476 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} servicekey stdout | 2023-01-17 05:36:05,533 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:36:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:15.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 05:36:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:15 UTC)" (scheduled at 2023-01-17 05:36:05.646354+00:00) notificationworker stdout | 2023-01-17 05:36:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 5, 647165), True, datetime.datetime(2023, 1, 17, 5, 36, 5, 647165), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:15 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:36:05,729 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:36:06,300 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:36:06,430 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:36:06,691 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:36:06,729 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:36:07,060 [231] [DEBUG] [app] Starting request: urn:request:2f00854c-f3d1-4a22-8ae0-7b78a1c9b0cc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:07,060 [235] [DEBUG] [app] Starting request: urn:request:e90e6f92-4408-42ab-a7d8-f93486920efe (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:07,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:07,065 [242] [DEBUG] [app] Starting request: urn:request:cb5a9dac-ab2c-44e9-8dc5-a5308cd138da (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:07,066 [243] [DEBUG] [app] Starting request: urn:request:cc74c791-8069-47a4-b175-602b4e596a2c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:07,066 [242] [DEBUG] [app] Ending request: urn:request:cb5a9dac-ab2c-44e9-8dc5-a5308cd138da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb5a9dac-ab2c-44e9-8dc5-a5308cd138da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:36:07,066 [243] [DEBUG] [app] Ending request: urn:request:cc74c791-8069-47a4-b175-602b4e596a2c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc74c791-8069-47a4-b175-602b4e596a2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:36:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:36:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:36:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:07,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:07,071 [235] [DEBUG] [app] Starting request: urn:request:d9b4426a-76ed-467e-b33b-80b2ff3acb62 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:07,071 [235] [DEBUG] [app] Ending request: urn:request:d9b4426a-76ed-467e-b33b-80b2ff3acb62 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9b4426a-76ed-467e-b33b-80b2ff3acb62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:07,071 [233] [DEBUG] [app] Starting request: urn:request:445065eb-cd47-4418-a88a-eeb16429e276 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:07,072 [233] [DEBUG] [app] Ending request: urn:request:445065eb-cd47-4418-a88a-eeb16429e276 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:445065eb-cd47-4418-a88a-eeb16429e276', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:36:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:07,087 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:07,089 [231] [DEBUG] [app] Ending request: urn:request:2f00854c-f3d1-4a22-8ae0-7b78a1c9b0cc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f00854c-f3d1-4a22-8ae0-7b78a1c9b0cc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:07,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:36:07,096 [235] [DEBUG] [app] Ending request: urn:request:e90e6f92-4408-42ab-a7d8-f93486920efe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e90e6f92-4408-42ab-a7d8-f93486920efe', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:07,096 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:07,097 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.038 47 0.037) globalpromstats stdout | 2023-01-17 05:36:07,308 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:36:07,871 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:36:07,892 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:36:08,174 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:36:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:36:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:40.738568+00:00 (in 29.999518 seconds) buildlogsarchiver stdout | 2023-01-17 05:36:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:40 UTC)" (scheduled at 2023-01-17 05:36:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:36:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 36, 10, 739419), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:36:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:36:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:36:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:40 UTC)" executed successfully builder stdout | 2023-01-17 05:36:12,063 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:12,075 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:12,075 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:12,085 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:25.646354+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 05:36:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:25 UTC)" (scheduled at 2023-01-17 05:36:15.646354+00:00) notificationworker stdout | 2023-01-17 05:36:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 15, 647193), True, datetime.datetime(2023, 1, 17, 5, 36, 15, 647193), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:36:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:19.537472+00:00 (in 2.600012 seconds) securityworker stdout | 2023-01-17 05:36:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:46 UTC)" (scheduled at 2023-01-17 05:36:16.936988+00:00) securityworker stdout | 2023-01-17 05:36:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:36:17,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:36:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:19.346120+00:00 (in 1.999976 seconds) repositorygcworker stdout | 2023-01-17 05:36:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:17 UTC)" (scheduled at 2023-01-17 05:36:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:36:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 36, 17, 346570), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:36:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 17, 346570), True, datetime.datetime(2023, 1, 17, 5, 36, 17, 346570), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:36:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 17, 346570), True, datetime.datetime(2023, 1, 17, 5, 36, 17, 346570), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 36, 17, 346570), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:36:17,365 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:36:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:36:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:36:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:04.936835+00:00 (in 45.590216 seconds) repositorygcworker stdout | 2023-01-17 05:36:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:19 UTC)" (scheduled at 2023-01-17 05:36:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:36:19,347 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:36:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:36:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:46.936988+00:00 (in 27.399002 seconds) securityworker stdout | 2023-01-17 05:36:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:49 UTC)" (scheduled at 2023-01-17 05:36:19.537472+00:00) securityworker stdout | 2023-01-17 05:36:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:36:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:36:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:36:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:36:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:36:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:36:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:36:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:36:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:51.348605+00:00 (in 29.999506 seconds) gcworker stdout | 2023-01-17 05:36:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:51 UTC)" (scheduled at 2023-01-17 05:36:21.348605+00:00) gcworker stdout | 2023-01-17 05:36:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724181349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:36:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:36:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:36:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:36:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:36:22,060 [233] [DEBUG] [app] Starting request: urn:request:eb76ac50-c595-4bf0-ab1c-fc25b372eb20 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:22,060 [236] [DEBUG] [app] Starting request: urn:request:77d0de36-15ad-4ed0-a792-a90d47ec3666 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:22,065 [243] [DEBUG] [app] Starting request: urn:request:4218b595-c44e-4fff-8bf1-01eb6041bb44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:22,065 [230] [DEBUG] [app] Starting request: urn:request:57d55e24-8fff-4233-ba4b-066fce19f3bd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:22,065 [243] [DEBUG] [app] Ending request: urn:request:4218b595-c44e-4fff-8bf1-01eb6041bb44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4218b595-c44e-4fff-8bf1-01eb6041bb44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:22,066 [230] [DEBUG] [app] Ending request: urn:request:57d55e24-8fff-4233-ba4b-066fce19f3bd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57d55e24-8fff-4233-ba4b-066fce19f3bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:36:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:36:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:22,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:22,070 [231] [DEBUG] [app] Starting request: urn:request:25f21d51-ec6f-4822-bb83-b2f9b8ad8c31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:22,070 [235] [DEBUG] [app] Starting request: urn:request:4c118a38-21d7-479e-a24b-9126b375e753 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:22,071 [231] [DEBUG] [app] Ending request: urn:request:25f21d51-ec6f-4822-bb83-b2f9b8ad8c31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25f21d51-ec6f-4822-bb83-b2f9b8ad8c31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:22,071 [235] [DEBUG] [app] Ending request: urn:request:4c118a38-21d7-479e-a24b-9126b375e753 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c118a38-21d7-479e-a24b-9126b375e753', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:36:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:22,088 [236] [DEBUG] [app] Ending request: urn:request:77d0de36-15ad-4ed0-a792-a90d47ec3666 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77d0de36-15ad-4ed0-a792-a90d47ec3666', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:22,088 [233] [DEBUG] [app] Ending request: urn:request:eb76ac50-c595-4bf0-ab1c-fc25b372eb20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb76ac50-c595-4bf0-ab1c-fc25b372eb20', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:22,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:36:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:36:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:36:22,096 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:22,107 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:22,107 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:22,118 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:36:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:36:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:37.256652+00:00 (in 13.719138 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:36:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:23 UTC)" (scheduled at 2023-01-17 05:36:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:36:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 36, 23, 537885), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:36:23,549 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 23, 537885), True, datetime.datetime(2023, 1, 17, 5, 36, 23, 537885), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:36:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 23, 537885), True, datetime.datetime(2023, 1, 17, 5, 36, 23, 537885), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 36, 23, 537885), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:36:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:36:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:35.646354+00:00 (in 9.999504 seconds) notificationworker stdout | 2023-01-17 05:36:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:35 UTC)" (scheduled at 2023-01-17 05:36:25.646354+00:00) notificationworker stdout | 2023-01-17 05:36:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 25, 647328), True, datetime.datetime(2023, 1, 17, 5, 36, 25, 647328), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:36:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:36:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:35.853396+00:00 (in 6.889312 seconds) exportactionlogsworker stdout | 2023-01-17 05:36:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:28 UTC)" (scheduled at 2023-01-17 05:36:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:36:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:36:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:29,103 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:36:29,199 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:36:30,036 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:36:30,541 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:36:30,572 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:36:30,916 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:36:31,098 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:36:31,129 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:36:31,131 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:36:31,135 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:36:31,140 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:36:31,142 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:36:32,128 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:32,140 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:32,140 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:32,151 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:36:34,255 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:36:34,358 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:36:35,118 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:36:35,120 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:36:35,122 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:36:35,455 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:36:35,457 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:36:35,465 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:36:35,468 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:36:35,473 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:36:35,477 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:36:35,479 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:36:35,481 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:36:35,484 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:36:35,486 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:36:35,540 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:36:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:45.646354+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:36:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:45 UTC)" (scheduled at 2023-01-17 05:36:35.646354+00:00) notificationworker stdout | 2023-01-17 05:36:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 35, 647296), True, datetime.datetime(2023, 1, 17, 5, 36, 35, 647296), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:45 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:36:35,746 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} exportactionlogsworker stdout | 2023-01-17 05:36:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:36:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:54.963413+00:00 (in 19.109551 seconds) exportactionlogsworker stdout | 2023-01-17 05:36:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:35 UTC)" (scheduled at 2023-01-17 05:36:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:36:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:36:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 35, 854338), True, datetime.datetime(2023, 1, 17, 5, 36, 35, 854338), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:36:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:36:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:36:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:36:36,311 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:36:36,439 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:36:36,727 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:36:36,759 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:36:37,060 [236] [DEBUG] [app] Starting request: urn:request:927024a7-e59e-4f34-9a9b-4d9a2991246f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:37,060 [231] [DEBUG] [app] Starting request: urn:request:86266bbd-a595-4851-95b1-55a163a2cf0c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:37,065 [243] [DEBUG] [app] Starting request: urn:request:b5a54387-9523-4b42-97e2-71615fe84334 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:37,065 [230] [DEBUG] [app] Starting request: urn:request:99da1050-e605-4ffc-bff4-8ece3f3fc396 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:37,065 [243] [DEBUG] [app] Ending request: urn:request:b5a54387-9523-4b42-97e2-71615fe84334 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b5a54387-9523-4b42-97e2-71615fe84334', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:36:37,066 [230] [DEBUG] [app] Ending request: urn:request:99da1050-e605-4ffc-bff4-8ece3f3fc396 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99da1050-e605-4ffc-bff4-8ece3f3fc396', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:36:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:36:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:37,070 [233] [DEBUG] [app] Starting request: urn:request:f854d832-4314-4687-87bd-f65b6d2a3dd5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:37,070 [236] [DEBUG] [app] Starting request: urn:request:5c05fa1a-91fb-41f8-ad05-c2678e54aff2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:37,071 [233] [DEBUG] [app] Ending request: urn:request:f854d832-4314-4687-87bd-f65b6d2a3dd5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f854d832-4314-4687-87bd-f65b6d2a3dd5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:37,071 [236] [DEBUG] [app] Ending request: urn:request:5c05fa1a-91fb-41f8-ad05-c2678e54aff2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5c05fa1a-91fb-41f8-ad05-c2678e54aff2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:36:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:37,088 [231] [DEBUG] [app] Ending request: urn:request:86266bbd-a595-4851-95b1-55a163a2cf0c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:86266bbd-a595-4851-95b1-55a163a2cf0c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:37,088 [236] [DEBUG] [app] Ending request: urn:request:927024a7-e59e-4f34-9a9b-4d9a2991246f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:927024a7-e59e-4f34-9a9b-4d9a2991246f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:36:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:36:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:36:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:36:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:37.537407+00:00 (in 0.280296 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:36:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:37 UTC)" (scheduled at 2023-01-17 05:36:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:36:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:36:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 37, 257563), True, datetime.datetime(2023, 1, 17, 5, 36, 37, 257563), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:36:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:36:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:36:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:36:37,331 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:36:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:36:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:37.256652+00:00 (in 59.718778 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:36:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:37 UTC)" (scheduled at 2023-01-17 05:36:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:36:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:36:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:37 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:36:37,905 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:36:37,923 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:36:38,206 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:36:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:36:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:10.738568+00:00 (in 29.999524 seconds) buildlogsarchiver stdout | 2023-01-17 05:36:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:10 UTC)" (scheduled at 2023-01-17 05:36:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:36:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 36, 40, 739451), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:36:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:36:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:36:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:36:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:36:41,149 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:42.941536+00:00 (in 1.792675 seconds) namespacegcworker stdout | 2023-01-17 05:36:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:41 UTC)" (scheduled at 2023-01-17 05:36:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:36:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:36:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 41, 149300), True, datetime.datetime(2023, 1, 17, 5, 36, 41, 149300), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:36:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:36:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:36:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:41 UTC)" executed successfully builder stdout | 2023-01-17 05:36:42,161 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:42,173 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:42,174 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:42,184 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:36:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:36:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:06.941185+00:00 (in 23.999162 seconds) namespacegcworker stdout | 2023-01-17 05:36:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:42 UTC)" (scheduled at 2023-01-17 05:36:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:36:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:36:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:37:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:55.646354+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:36:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:55 UTC)" (scheduled at 2023-01-17 05:36:45.646354+00:00) notificationworker stdout | 2023-01-17 05:36:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 45, 647334), True, datetime.datetime(2023, 1, 17, 5, 36, 45, 647334), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:36:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:36:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:36:49.537472+00:00 (in 2.599981 seconds) securityworker stdout | 2023-01-17 05:36:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:16 UTC)" (scheduled at 2023-01-17 05:36:46.936988+00:00) securityworker stdout | 2023-01-17 05:36:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:36:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:36:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:16.936988+00:00 (in 27.399007 seconds) securityworker stdout | 2023-01-17 05:36:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:19 UTC)" (scheduled at 2023-01-17 05:36:49.537472+00:00) securityworker stdout | 2023-01-17 05:36:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:36:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:36:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:36:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:36:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:36:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:36:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:36:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:36:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:36:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:36:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:36:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:36:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:36:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:36:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:36:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:21.348605+00:00 (in 29.999492 seconds) gcworker stdout | 2023-01-17 05:36:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:21 UTC)" (scheduled at 2023-01-17 05:36:51.348605+00:00) gcworker stdout | 2023-01-17 05:36:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724211349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:36:51,363 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:36:51,363 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:36:51,363 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:36:52,060 [235] [DEBUG] [app] Starting request: urn:request:bb2bb065-8e8a-4947-9c05-7dca03cf0c66 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:52,060 [231] [DEBUG] [app] Starting request: urn:request:ce18edbd-1dd6-4822-8c1c-a199c2d2a828 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:36:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:36:52,065 [243] [DEBUG] [app] Starting request: urn:request:36effc40-ab75-4724-bf12-9ce1c55ed130 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,065 [242] [DEBUG] [app] Starting request: urn:request:01166729-7460-4912-b72a-93cd303e628c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,066 [243] [DEBUG] [app] Ending request: urn:request:36effc40-ab75-4724-bf12-9ce1c55ed130 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36effc40-ab75-4724-bf12-9ce1c55ed130', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,066 [242] [DEBUG] [app] Ending request: urn:request:01166729-7460-4912-b72a-93cd303e628c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01166729-7460-4912-b72a-93cd303e628c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:36:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:36:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:36:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:36:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:36:52,070 [236] [DEBUG] [app] Starting request: urn:request:21e4e34f-28e4-4bca-9faf-ada8bea59b17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:52,070 [233] [DEBUG] [app] Starting request: urn:request:b8652ba0-0930-47b7-b8f7-160392a71051 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:36:52,070 [236] [DEBUG] [app] Ending request: urn:request:21e4e34f-28e4-4bca-9faf-ada8bea59b17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21e4e34f-28e4-4bca-9faf-ada8bea59b17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:36:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:36:52,071 [233] [DEBUG] [app] Ending request: urn:request:b8652ba0-0930-47b7-b8f7-160392a71051 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8652ba0-0930-47b7-b8f7-160392a71051', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:36:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:36:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:36:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:36:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:36:52,087 [235] [DEBUG] [app] Ending request: urn:request:bb2bb065-8e8a-4947-9c05-7dca03cf0c66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb2bb065-8e8a-4947-9c05-7dca03cf0c66', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:52,087 [231] [DEBUG] [app] Ending request: urn:request:ce18edbd-1dd6-4822-8c1c-a199c2d2a828 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce18edbd-1dd6-4822-8c1c-a199c2d2a828', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:36:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:36:52,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:36:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:36:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:36:52,194 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:36:52,206 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:36:52,206 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:36:52,217 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:36:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:36:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:28.963662+00:00 (in 33.999714 seconds) exportactionlogsworker stdout | 2023-01-17 05:36:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:54 UTC)" (scheduled at 2023-01-17 05:36:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:36:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 36, 54, 964401), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:36:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 54, 964401), True, datetime.datetime(2023, 1, 17, 5, 36, 54, 964401), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:36:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 36, 54, 964401), True, datetime.datetime(2023, 1, 17, 5, 36, 54, 964401), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 36, 54, 964401), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:36:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:36:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:41:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:36:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:01.636362+00:00 (in 5.989484 seconds) notificationworker stdout | 2023-01-17 05:36:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:05 UTC)" (scheduled at 2023-01-17 05:36:55.646354+00:00) notificationworker stdout | 2023-01-17 05:36:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:36:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 36, 55, 647284), True, datetime.datetime(2023, 1, 17, 5, 36, 55, 647284), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:36:55,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:36:55,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:36:55,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:36:59,137 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:36:59,234 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:37:00,070 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:37:00,577 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:37:00,608 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:37:00,925 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:37:01,134 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 05:37:01,136 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:37:01,138 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:37:01,141 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:37:01,146 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:37:01,148 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:37:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:05.646354+00:00 (in 4.009549 seconds) notificationworker stdout | 2023-01-17 05:37:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:01 UTC)" (scheduled at 2023-01-17 05:37:01.636362+00:00) notificationworker stdout | 2023-01-17 05:37:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:37:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:01 UTC)" executed successfully builder stdout | 2023-01-17 05:37:02,227 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:02,240 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:02,240 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:02,250 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:37:04,291 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:37:04,393 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:37:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:37:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:19.346120+00:00 (in 14.408705 seconds) repositorygcworker stdout | 2023-01-17 05:37:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:04 UTC)" (scheduled at 2023-01-17 05:37:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:37:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:37:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 4, 937814), True, datetime.datetime(2023, 1, 17, 5, 37, 4, 937814), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:37:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:37:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:37:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:37:05,124 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:37:05,127 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:37:05,129 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:37:05,466 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:37:05,469 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:37:05,474 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:37:05,477 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:37:05,481 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:37:05,484 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:37:05,488 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:37:05,490 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:37:05,493 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:37:05,509 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:37:05,575 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:37:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:15.646354+00:00 (in 9.999562 seconds) notificationworker stdout | 2023-01-17 05:37:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:15 UTC)" (scheduled at 2023-01-17 05:37:05.646354+00:00) notificationworker stdout | 2023-01-17 05:37:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 5, 647175), True, datetime.datetime(2023, 1, 17, 5, 37, 5, 647175), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:15 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:37:05,757 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:37:06,346 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:37:06,468 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:37:06,739 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:37:06,795 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} namespacegcworker stdout | 2023-01-17 05:37:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:37:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:41.148245+00:00 (in 34.206610 seconds) namespacegcworker stdout | 2023-01-17 05:37:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:42:06 UTC)" (scheduled at 2023-01-17 05:37:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:37:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 37, 6, 942024), 'namespacegc/%']) manifestbackfillworker stdout | 2023-01-17 05:37:06,951 [72] [DEBUG] [apscheduler.scheduler] Looking for jobs to run manifestbackfillworker stdout | 2023-01-17 05:37:06,951 [72] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2023-01-17 05:37:06 UTC)" (scheduled at 2023-01-17 05:37:06.951421+00:00) manifestbackfillworker stdout | 2023-01-17 05:37:06,952 [72] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0]) manifestbackfillworker stdout | 2023-01-17 05:37:06,952 [72] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:37:06.951421+00:00 (in 3599.999714 seconds) namespacegcworker stdout | 2023-01-17 05:37:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 37, 6, 942024), True, datetime.datetime(2023, 1, 17, 5, 37, 6, 942024), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:37:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 37, 6, 942024), True, datetime.datetime(2023, 1, 17, 5, 37, 6, 942024), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 37, 6, 942024), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:37:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:37:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:42:06 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:37:06,962 [72] [DEBUG] [__main__] Manifest backfill worker has completed; skipping manifestbackfillworker stdout | 2023-01-17 05:37:06,962 [72] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 05:37:06,962 [72] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2023-01-17 06:37:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:37:07,060 [235] [DEBUG] [app] Starting request: urn:request:789d0b74-391a-4456-bbb0-dcb013d3f1b7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:07,060 [231] [DEBUG] [app] Starting request: urn:request:906ec194-9809-463e-b3b2-cd06eea24c93 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:07,065 [243] [DEBUG] [app] Starting request: urn:request:96473598-502e-4ac8-b66e-93cda3d200d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:07,065 [241] [DEBUG] [app] Starting request: urn:request:cbac6c8e-8a0e-48c4-9e71-58bcb0b2722c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:07,066 [243] [DEBUG] [app] Ending request: urn:request:96473598-502e-4ac8-b66e-93cda3d200d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96473598-502e-4ac8-b66e-93cda3d200d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:37:07,066 [241] [DEBUG] [app] Ending request: urn:request:cbac6c8e-8a0e-48c4-9e71-58bcb0b2722c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cbac6c8e-8a0e-48c4-9e71-58bcb0b2722c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:37:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:37:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:07,070 [236] [DEBUG] [app] Starting request: urn:request:192d991e-8dde-4572-9f98-0a5dddee3de7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:07,070 [233] [DEBUG] [app] Starting request: urn:request:5b8581f2-18de-4819-9182-b84562a09707 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:07,071 [236] [DEBUG] [app] Ending request: urn:request:192d991e-8dde-4572-9f98-0a5dddee3de7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:192d991e-8dde-4572-9f98-0a5dddee3de7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:07,071 [233] [DEBUG] [app] Ending request: urn:request:5b8581f2-18de-4819-9182-b84562a09707 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5b8581f2-18de-4819-9182-b84562a09707', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:37:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:07,087 [231] [DEBUG] [app] Ending request: urn:request:906ec194-9809-463e-b3b2-cd06eea24c93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:906ec194-9809-463e-b3b2-cd06eea24c93', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:07,088 [235] [DEBUG] [app] Ending request: urn:request:789d0b74-391a-4456-bbb0-dcb013d3f1b7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:789d0b74-391a-4456-bbb0-dcb013d3f1b7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:37:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) globalpromstats stdout | 2023-01-17 05:37:07,338 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:37:07,939 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:37:07,956 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:37:08,242 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:37:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:37:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:40.738568+00:00 (in 29.999520 seconds) buildlogsarchiver stdout | 2023-01-17 05:37:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:40 UTC)" (scheduled at 2023-01-17 05:37:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:37:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 37, 10, 739457), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:37:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:37:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:37:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:40 UTC)" executed successfully builder stdout | 2023-01-17 05:37:12,251 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:12,263 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:12,263 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:12,274 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:25.646354+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 05:37:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:25 UTC)" (scheduled at 2023-01-17 05:37:15.646354+00:00) notificationworker stdout | 2023-01-17 05:37:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 15, 647337), True, datetime.datetime(2023, 1, 17, 5, 37, 15, 647337), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:37:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:19.537472+00:00 (in 2.599996 seconds) securityworker stdout | 2023-01-17 05:37:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:46 UTC)" (scheduled at 2023-01-17 05:37:16.936988+00:00) securityworker stdout | 2023-01-17 05:37:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:37:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:37:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:04.936835+00:00 (in 45.590288 seconds) repositorygcworker stdout | 2023-01-17 05:37:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:19 UTC)" (scheduled at 2023-01-17 05:37:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:37:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:37:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:37:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:46.936988+00:00 (in 27.399030 seconds) securityworker stdout | 2023-01-17 05:37:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:49 UTC)" (scheduled at 2023-01-17 05:37:19.537472+00:00) securityworker stdout | 2023-01-17 05:37:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:37:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:37:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:37:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:37:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:37:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:37:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:37:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:37:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:51.348605+00:00 (in 29.999482 seconds) gcworker stdout | 2023-01-17 05:37:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:51 UTC)" (scheduled at 2023-01-17 05:37:21.348605+00:00) gcworker stdout | 2023-01-17 05:37:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724241349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:37:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:37:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:37:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:37:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:37:22,062 [235] [DEBUG] [app] Starting request: urn:request:afa27d67-a5f7-45cc-8aef-fbb26f161a0a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:22,062 [236] [DEBUG] [app] Starting request: urn:request:9b542468-775f-486c-9d8a-7c0eb4ca8c72 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:22,064 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,064 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,066 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:22,067 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:22,067 [242] [DEBUG] [app] Starting request: urn:request:ab05c2d5-5cd9-4c1f-8bf8-8a1a11b904c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:22,068 [230] [DEBUG] [app] Starting request: urn:request:3f07ba46-e01e-4fe9-8ecc-0bb8121a3275 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:22,068 [242] [DEBUG] [app] Ending request: urn:request:ab05c2d5-5cd9-4c1f-8bf8-8a1a11b904c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab05c2d5-5cd9-4c1f-8bf8-8a1a11b904c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:22,068 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:37:22,068 [230] [DEBUG] [app] Ending request: urn:request:3f07ba46-e01e-4fe9-8ecc-0bb8121a3275 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f07ba46-e01e-4fe9-8ecc-0bb8121a3275', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:37:22,068 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:37:22,068 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:22,068 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:37:22,069 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,070 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:22,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:22,072 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:22,072 [231] [DEBUG] [app] Starting request: urn:request:6bb6e485-a0e5-4e9c-85ea-7dfebdc62158 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:22,072 [233] [DEBUG] [app] Starting request: urn:request:57eeb3b4-aa5d-4d64-b07c-7c8065dcc1a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:22,072 [231] [DEBUG] [app] Ending request: urn:request:6bb6e485-a0e5-4e9c-85ea-7dfebdc62158 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6bb6e485-a0e5-4e9c-85ea-7dfebdc62158', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:22,073 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:22,073 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:22,073 [233] [DEBUG] [app] Ending request: urn:request:57eeb3b4-aa5d-4d64-b07c-7c8065dcc1a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57eeb3b4-aa5d-4d64-b07c-7c8065dcc1a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:37:22,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:22,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:22,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:22,074 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:22,083 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:22,089 [235] [DEBUG] [app] Ending request: urn:request:afa27d67-a5f7-45cc-8aef-fbb26f161a0a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:afa27d67-a5f7-45cc-8aef-fbb26f161a0a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:22,090 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:22,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:37:22,092 [236] [DEBUG] [app] Ending request: urn:request:9b542468-775f-486c-9d8a-7c0eb4ca8c72 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b542468-775f-486c-9d8a-7c0eb4ca8c72', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:22,092 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:22,093 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.031) builder stdout | 2023-01-17 05:37:22,284 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:22,296 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:22,297 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:22,307 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:35.646354+00:00 (in 9.999434 seconds) notificationworker stdout | 2023-01-17 05:37:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:35 UTC)" (scheduled at 2023-01-17 05:37:25.646354+00:00) notificationworker stdout | 2023-01-17 05:37:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 25, 647447), True, datetime.datetime(2023, 1, 17, 5, 37, 25, 647447), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:37:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:37:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:35.853396+00:00 (in 6.889206 seconds) exportactionlogsworker stdout | 2023-01-17 05:37:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:28 UTC)" (scheduled at 2023-01-17 05:37:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:37:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:37:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:29,166 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:37:29,240 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:37:30,106 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:37:30,601 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:37:30,642 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:37:30,940 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:37:31,143 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:37:31,145 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:37:31,147 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:37:31,153 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:37:31,155 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:37:31,157 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:37:32,317 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:32,330 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:32,330 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:32,341 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:37:34,325 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:37:34,427 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:37:35,131 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:37:35,134 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:37:35,136 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:37:35,476 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:37:35,478 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:37:35,482 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:37:35,485 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:37:35,489 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:37:35,491 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:37:35,495 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:37:35,499 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:37:35,501 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:37:35,531 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:37:35,609 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:37:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:45.646354+00:00 (in 9.999466 seconds) notificationworker stdout | 2023-01-17 05:37:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:45 UTC)" (scheduled at 2023-01-17 05:37:35.646354+00:00) notificationworker stdout | 2023-01-17 05:37:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 35, 647372), True, datetime.datetime(2023, 1, 17, 5, 37, 35, 647372), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:45 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:37:35,792 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} exportactionlogsworker stdout | 2023-01-17 05:37:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:37:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:28.963662+00:00 (in 53.109799 seconds) exportactionlogsworker stdout | 2023-01-17 05:37:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:35 UTC)" (scheduled at 2023-01-17 05:37:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:37:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:37:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 35, 854303), True, datetime.datetime(2023, 1, 17, 5, 37, 35, 854303), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:37:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:37:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:37:35,866 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:37:36,352 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:37:36,504 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:37:36,774 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:37:36,805 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:37:37,060 [236] [DEBUG] [app] Starting request: urn:request:d655259a-e5d9-40e9-96c0-950c87128615 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:37,060 [235] [DEBUG] [app] Starting request: urn:request:900a0867-d3cb-42d8-af36-d3116f7c06c4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:37,065 [242] [DEBUG] [app] Starting request: urn:request:4535a282-fab1-4618-9bbd-c7a9ccc64dc4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:37,065 [230] [DEBUG] [app] Starting request: urn:request:454f986a-2e27-4ff8-a365-17bcc300597c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:37,065 [242] [DEBUG] [app] Ending request: urn:request:4535a282-fab1-4618-9bbd-c7a9ccc64dc4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4535a282-fab1-4618-9bbd-c7a9ccc64dc4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:37:37,066 [230] [DEBUG] [app] Ending request: urn:request:454f986a-2e27-4ff8-a365-17bcc300597c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:454f986a-2e27-4ff8-a365-17bcc300597c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:37:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:37:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:37,070 [233] [DEBUG] [app] Starting request: urn:request:f6e5b882-d78f-4c96-8add-c055d537efb4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:37,070 [236] [DEBUG] [app] Starting request: urn:request:fb0239f5-338d-4ca8-a753-0b02b0a0e850 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:37,071 [233] [DEBUG] [app] Ending request: urn:request:f6e5b882-d78f-4c96-8add-c055d537efb4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6e5b882-d78f-4c96-8add-c055d537efb4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:37,071 [236] [DEBUG] [app] Ending request: urn:request:fb0239f5-338d-4ca8-a753-0b02b0a0e850 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fb0239f5-338d-4ca8-a753-0b02b0a0e850', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:37,088 [235] [DEBUG] [app] Ending request: urn:request:900a0867-d3cb-42d8-af36-d3116f7c06c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:900a0867-d3cb-42d8-af36-d3116f7c06c4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:37:37,088 [236] [DEBUG] [app] Ending request: urn:request:d655259a-e5d9-40e9-96c0-950c87128615 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d655259a-e5d9-40e9-96c0-950c87128615', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:37:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:37:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:37:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:37.537407+00:00 (in 0.280313 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:37:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:37 UTC)" (scheduled at 2023-01-17 05:37:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:37:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:37:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 37, 257509), True, datetime.datetime(2023, 1, 17, 5, 37, 37, 257509), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:37:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:37:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:37:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:37:37,358 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:37:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:37:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:37.256652+00:00 (in 59.718805 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:37:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:37 UTC)" (scheduled at 2023-01-17 05:37:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:37:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:37:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:37 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:37:37,975 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:37:37,981 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:37:38,249 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:37:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:37:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:10.738568+00:00 (in 29.999513 seconds) buildlogsarchiver stdout | 2023-01-17 05:37:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:10 UTC)" (scheduled at 2023-01-17 05:37:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:37:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 37, 40, 739465), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:37:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:37:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:37:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:37:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:37:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:42.941536+00:00 (in 1.792801 seconds) namespacegcworker stdout | 2023-01-17 05:37:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:41 UTC)" (scheduled at 2023-01-17 05:37:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:37:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:37:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 41, 149132), True, datetime.datetime(2023, 1, 17, 5, 37, 41, 149132), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:37:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:37:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:37:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:41 UTC)" executed successfully builder stdout | 2023-01-17 05:37:42,346 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:42,359 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:42,359 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:42,370 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:37:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:37:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:41.148245+00:00 (in 58.206232 seconds) namespacegcworker stdout | 2023-01-17 05:37:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:42 UTC)" (scheduled at 2023-01-17 05:37:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:37:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:37:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:38:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:55.646354+00:00 (in 9.999455 seconds) notificationworker stdout | 2023-01-17 05:37:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:55 UTC)" (scheduled at 2023-01-17 05:37:45.646354+00:00) notificationworker stdout | 2023-01-17 05:37:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 45, 647397), True, datetime.datetime(2023, 1, 17, 5, 37, 45, 647397), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:37:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:37:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:37:49.537472+00:00 (in 2.599995 seconds) securityworker stdout | 2023-01-17 05:37:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:16 UTC)" (scheduled at 2023-01-17 05:37:46.936988+00:00) securityworker stdout | 2023-01-17 05:37:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:37:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:37:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:16.936988+00:00 (in 27.399000 seconds) securityworker stdout | 2023-01-17 05:37:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:19 UTC)" (scheduled at 2023-01-17 05:37:49.537472+00:00) securityworker stdout | 2023-01-17 05:37:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:37:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:37:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:37:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:37:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:37:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:37:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:37:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:37:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:37:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:37:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:37:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:37:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:37:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:37:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:37:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:21.348605+00:00 (in 29.999487 seconds) gcworker stdout | 2023-01-17 05:37:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:21 UTC)" (scheduled at 2023-01-17 05:37:51.348605+00:00) gcworker stdout | 2023-01-17 05:37:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724271349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:37:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:37:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:37:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:37:52,060 [231] [DEBUG] [app] Starting request: urn:request:351041b9-661c-4052-a160-0c9359f438ec (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:52,060 [233] [DEBUG] [app] Starting request: urn:request:06bd373d-d813-4283-ab96-40a29e8205f8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:37:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:37:52,065 [243] [DEBUG] [app] Starting request: urn:request:23875331-814e-4581-ad7c-2e7aa9547d12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:52,065 [243] [DEBUG] [app] Ending request: urn:request:23875331-814e-4581-ad7c-2e7aa9547d12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23875331-814e-4581-ad7c-2e7aa9547d12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:37:52,065 [242] [DEBUG] [app] Starting request: urn:request:99611ae1-1bcd-4265-b1f4-4e6da91cfc17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:37:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:37:52,066 [242] [DEBUG] [app] Ending request: urn:request:99611ae1-1bcd-4265-b1f4-4e6da91cfc17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99611ae1-1bcd-4265-b1f4-4e6da91cfc17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:37:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:37:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:37:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:37:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:52,069 [231] [DEBUG] [app] Starting request: urn:request:8217b94b-d327-42c0-9f5c-40fe0def4cf7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:52,070 [231] [DEBUG] [app] Ending request: urn:request:8217b94b-d327-42c0-9f5c-40fe0def4cf7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8217b94b-d327-42c0-9f5c-40fe0def4cf7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:52,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:37:52,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:52,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:37:52,072 [236] [DEBUG] [app] Starting request: urn:request:29286fc9-8012-40cd-ba0e-2d42e73e8a17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:37:52,072 [236] [DEBUG] [app] Ending request: urn:request:29286fc9-8012-40cd-ba0e-2d42e73e8a17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:29286fc9-8012-40cd-ba0e-2d42e73e8a17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:37:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:37:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:37:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:37:52,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:37:52,087 [233] [DEBUG] [app] Ending request: urn:request:06bd373d-d813-4283-ab96-40a29e8205f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:06bd373d-d813-4283-ab96-40a29e8205f8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:52,087 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:37:52,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:37:52,088 [231] [DEBUG] [app] Ending request: urn:request:351041b9-661c-4052-a160-0c9359f438ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:351041b9-661c-4052-a160-0c9359f438ec', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:37:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:37:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:37:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) builder stdout | 2023-01-17 05:37:52,374 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:37:52,386 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:37:52,386 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:37:52,397 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:37:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:01.636362+00:00 (in 5.989439 seconds) notificationworker stdout | 2023-01-17 05:37:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:05 UTC)" (scheduled at 2023-01-17 05:37:55.646354+00:00) notificationworker stdout | 2023-01-17 05:37:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:37:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 37, 55, 647416), True, datetime.datetime(2023, 1, 17, 5, 37, 55, 647416), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:37:55,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:37:55,662 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:37:55,662 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:37:59,199 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:37:59,266 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:38:00,125 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:38:00,636 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:38:00,655 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:38:00,952 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:38:01,149 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:38:01,153 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:38:01,155 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:38:01,158 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:38:01,160 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:38:01,186 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:38:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:05.646354+00:00 (in 4.009439 seconds) notificationworker stdout | 2023-01-17 05:38:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:01 UTC)" (scheduled at 2023-01-17 05:38:01.636362+00:00) notificationworker stdout | 2023-01-17 05:38:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:38:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:01 UTC)" executed successfully builder stdout | 2023-01-17 05:38:02,407 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:02,420 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:02,420 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:02,430 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:38:04,348 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:38:04,463 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:38:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:38:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:19.346120+00:00 (in 14.408783 seconds) repositorygcworker stdout | 2023-01-17 05:38:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:04 UTC)" (scheduled at 2023-01-17 05:38:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:38:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:38:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 4, 937737), True, datetime.datetime(2023, 1, 17, 5, 38, 4, 937737), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:38:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:38:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:38:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:38:05,139 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:38:05,141 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:38:05,143 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:38:05,485 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:38:05,487 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:38:05,490 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:38:05,492 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:38:05,495 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:38:05,500 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:38:05,502 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:38:05,506 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:38:05,508 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:38:05,543 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:38:05,634 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:38:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:15.646354+00:00 (in 9.999356 seconds) notificationworker stdout | 2023-01-17 05:38:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:15 UTC)" (scheduled at 2023-01-17 05:38:05.646354+00:00) notificationworker stdout | 2023-01-17 05:38:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 5, 647457), True, datetime.datetime(2023, 1, 17, 5, 38, 5, 647457), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:15 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:38:05,810 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:38:06,366 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:38:06,539 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:38:06,800 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:38:06,839 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:38:07,060 [236] [DEBUG] [app] Starting request: urn:request:bb964b65-83b5-4e36-a43d-a3ee653229fa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:07,060 [231] [DEBUG] [app] Starting request: urn:request:2cd62a35-d82e-4f48-aca5-7fb0f80554a0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:07,065 [230] [DEBUG] [app] Starting request: urn:request:e7063808-b54b-436f-bf9e-b9eadb596ae7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:07,065 [243] [DEBUG] [app] Starting request: urn:request:858ff5d2-95b8-433f-a79f-bc3164c56d96 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:07,065 [230] [DEBUG] [app] Ending request: urn:request:e7063808-b54b-436f-bf9e-b9eadb596ae7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e7063808-b54b-436f-bf9e-b9eadb596ae7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:38:07,066 [243] [DEBUG] [app] Ending request: urn:request:858ff5d2-95b8-433f-a79f-bc3164c56d96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:858ff5d2-95b8-433f-a79f-bc3164c56d96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:38:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:38:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:07,070 [233] [DEBUG] [app] Starting request: urn:request:37d66a5e-d317-4fa4-b5cb-0d97856bab05 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:07,070 [236] [DEBUG] [app] Starting request: urn:request:2f4b1750-b819-45f2-9403-184995bd1afd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:07,070 [233] [DEBUG] [app] Ending request: urn:request:37d66a5e-d317-4fa4-b5cb-0d97856bab05 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:37d66a5e-d317-4fa4-b5cb-0d97856bab05', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:07,070 [236] [DEBUG] [app] Ending request: urn:request:2f4b1750-b819-45f2-9403-184995bd1afd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f4b1750-b819-45f2-9403-184995bd1afd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:07,087 [236] [DEBUG] [app] Ending request: urn:request:bb964b65-83b5-4e36-a43d-a3ee653229fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb964b65-83b5-4e36-a43d-a3ee653229fa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:38:07,101 [231] [DEBUG] [app] Ending request: urn:request:2cd62a35-d82e-4f48-aca5-7fb0f80554a0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2cd62a35-d82e-4f48-aca5-7fb0f80554a0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:07,102 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:07,102 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.043 47 0.042) globalpromstats stdout | 2023-01-17 05:38:07,393 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:38:07,984 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:38:07,988 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:38:08,277 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:38:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:38:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:40.738568+00:00 (in 29.999512 seconds) buildlogsarchiver stdout | 2023-01-17 05:38:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:40 UTC)" (scheduled at 2023-01-17 05:38:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:38:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 38, 10, 739457), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:38:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:38:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:38:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:40 UTC)" executed successfully builder stdout | 2023-01-17 05:38:12,441 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:12,453 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:12,453 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:12,463 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:25.646354+00:00 (in 9.999486 seconds) notificationworker stdout | 2023-01-17 05:38:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:25 UTC)" (scheduled at 2023-01-17 05:38:15.646354+00:00) notificationworker stdout | 2023-01-17 05:38:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 15, 647275), True, datetime.datetime(2023, 1, 17, 5, 38, 15, 647275), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:38:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:19.537472+00:00 (in 2.600018 seconds) securityworker stdout | 2023-01-17 05:38:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:46 UTC)" (scheduled at 2023-01-17 05:38:16.936988+00:00) securityworker stdout | 2023-01-17 05:38:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:38:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:38:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:04.936835+00:00 (in 45.590266 seconds) repositorygcworker stdout | 2023-01-17 05:38:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:19 UTC)" (scheduled at 2023-01-17 05:38:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:38:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:38:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:38:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:46.936988+00:00 (in 27.399035 seconds) securityworker stdout | 2023-01-17 05:38:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:49 UTC)" (scheduled at 2023-01-17 05:38:19.537472+00:00) securityworker stdout | 2023-01-17 05:38:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:38:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:38:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:38:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:38:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:38:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:38:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:38:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:38:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:51.348605+00:00 (in 29.999522 seconds) gcworker stdout | 2023-01-17 05:38:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:51 UTC)" (scheduled at 2023-01-17 05:38:21.348605+00:00) gcworker stdout | 2023-01-17 05:38:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724301349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:38:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:38:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:38:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:38:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:38:22,060 [236] [DEBUG] [app] Starting request: urn:request:0576ec5d-7eaf-45fb-940e-4626d60ed29a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:22,060 [235] [DEBUG] [app] Starting request: urn:request:abb1b68c-4288-492b-b2c9-b8bb640926fd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,063 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:22,065 [230] [DEBUG] [app] Starting request: urn:request:69a44db7-2245-484b-b765-a07574428034 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:22,065 [230] [DEBUG] [app] Ending request: urn:request:69a44db7-2245-484b-b765-a07574428034 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:69a44db7-2245-484b-b765-a07574428034', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:38:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:38:22,066 [243] [DEBUG] [app] Starting request: urn:request:3a63dec5-a83f-4833-9207-778b11b11ee9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:22,066 [243] [DEBUG] [app] Ending request: urn:request:3a63dec5-a83f-4833-9207-778b11b11ee9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a63dec5-a83f-4833-9207-778b11b11ee9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:38:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:22,070 [233] [DEBUG] [app] Starting request: urn:request:4449ca53-07bf-4ee1-92d2-8960a0db0c73 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:22,070 [233] [DEBUG] [app] Ending request: urn:request:4449ca53-07bf-4ee1-92d2-8960a0db0c73 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4449ca53-07bf-4ee1-92d2-8960a0db0c73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:22,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:22,072 [231] [DEBUG] [app] Starting request: urn:request:d9dd5154-c9df-46eb-abd8-e08488b7a679 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:22,073 [231] [DEBUG] [app] Ending request: urn:request:d9dd5154-c9df-46eb-abd8-e08488b7a679 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9dd5154-c9df-46eb-abd8-e08488b7a679', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:38:22,073 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:22,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:22,074 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:22,087 [236] [DEBUG] [app] Ending request: urn:request:0576ec5d-7eaf-45fb-940e-4626d60ed29a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0576ec5d-7eaf-45fb-940e-4626d60ed29a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:38:22,090 [235] [DEBUG] [app] Ending request: urn:request:abb1b68c-4288-492b-b2c9-b8bb640926fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:abb1b68c-4288-492b-b2c9-b8bb640926fd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:22,090 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:22,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) builder stdout | 2023-01-17 05:38:22,473 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:22,486 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:22,486 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:22,496 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:35.646354+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 05:38:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:35 UTC)" (scheduled at 2023-01-17 05:38:25.646354+00:00) notificationworker stdout | 2023-01-17 05:38:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 25, 647241), True, datetime.datetime(2023, 1, 17, 5, 38, 25, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:38:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:38:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:35.853396+00:00 (in 6.889281 seconds) exportactionlogsworker stdout | 2023-01-17 05:38:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:28 UTC)" (scheduled at 2023-01-17 05:38:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:38:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:38:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:29,235 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:38:29,286 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:38:30,157 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:38:30,672 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:38:30,687 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:38:30,987 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:38:31,156 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:38:31,158 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:38:31,161 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:38:31,163 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:38:31,166 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:38:31,221 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:38:32,506 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:32,519 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:32,519 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:32,530 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:38:34,384 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:38:34,499 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:38:35,146 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:38:35,148 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:38:35,150 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:38:35,494 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:38:35,497 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:38:35,500 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:38:35,502 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:38:35,505 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:38:35,507 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:38:35,509 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:38:35,514 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:38:35,516 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:38:35,578 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:38:35,646 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:38:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:36.635997+00:00 (in 0.989094 seconds) notificationworker stdout | 2023-01-17 05:38:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:45 UTC)" (scheduled at 2023-01-17 05:38:35.646354+00:00) notificationworker stdout | 2023-01-17 05:38:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 35, 647378), True, datetime.datetime(2023, 1, 17, 5, 38, 35, 647378), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:45 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:38:35,844 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} exportactionlogsworker stdout | 2023-01-17 05:38:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:38:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:28.963662+00:00 (in 53.109789 seconds) exportactionlogsworker stdout | 2023-01-17 05:38:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:35 UTC)" (scheduled at 2023-01-17 05:38:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:38:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:38:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 35, 854275), True, datetime.datetime(2023, 1, 17, 5, 38, 35, 854275), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:38:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:38:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:38:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:35 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:38:36,401 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:38:36,569 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} notificationworker stdout | 2023-01-17 05:38:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:45.646354+00:00 (in 9.009852 seconds) notificationworker stdout | 2023-01-17 05:38:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:43:36 UTC)" (scheduled at 2023-01-17 05:38:36.635997+00:00) notificationworker stdout | 2023-01-17 05:38:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 38, 36, 636939), 'notification/%']) notificationworker stdout | 2023-01-17 05:38:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 38, 36, 636939), True, datetime.datetime(2023, 1, 17, 5, 38, 36, 636939), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:38:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 38, 36, 636939), True, datetime.datetime(2023, 1, 17, 5, 38, 36, 636939), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 38, 36, 636939), 'notification/%']) notificationworker stdout | 2023-01-17 05:38:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:43:36 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:38:36,836 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:38:36,870 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:38:37,060 [236] [DEBUG] [app] Starting request: urn:request:bf9c82a9-3d56-4b6b-8e31-ed4a58848a10 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:37,060 [235] [DEBUG] [app] Starting request: urn:request:834c2e7d-3c94-470a-b69b-16427fe030f6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:37,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:37,065 [230] [DEBUG] [app] Starting request: urn:request:4ed9718d-f824-4b28-bb3a-72a4c35450c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:37,066 [243] [DEBUG] [app] Starting request: urn:request:c2cfd49e-1213-4081-9567-2149100542e7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:37,066 [230] [DEBUG] [app] Ending request: urn:request:4ed9718d-f824-4b28-bb3a-72a4c35450c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ed9718d-f824-4b28-bb3a-72a4c35450c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:38:37,066 [243] [DEBUG] [app] Ending request: urn:request:c2cfd49e-1213-4081-9567-2149100542e7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2cfd49e-1213-4081-9567-2149100542e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:38:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:38:37,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:37,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:38:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:37,071 [231] [DEBUG] [app] Starting request: urn:request:49c755f9-926c-40d0-af4f-727a0c80c797 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:37,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:37,071 [231] [DEBUG] [app] Ending request: urn:request:49c755f9-926c-40d0-af4f-727a0c80c797 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:49c755f9-926c-40d0-af4f-727a0c80c797', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:37,072 [233] [DEBUG] [app] Starting request: urn:request:35ad7aac-6132-4405-a5e5-da276d83f5e6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:37,072 [233] [DEBUG] [app] Ending request: urn:request:35ad7aac-6132-4405-a5e5-da276d83f5e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:35ad7aac-6132-4405-a5e5-da276d83f5e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:37,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:37,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:37,084 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:37,085 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:37,093 [235] [DEBUG] [app] Ending request: urn:request:834c2e7d-3c94-470a-b69b-16427fe030f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:834c2e7d-3c94-470a-b69b-16427fe030f6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:37,094 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:37,094 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.034) gunicorn-web stdout | 2023-01-17 05:38:37,094 [236] [DEBUG] [app] Ending request: urn:request:bf9c82a9-3d56-4b6b-8e31-ed4a58848a10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf9c82a9-3d56-4b6b-8e31-ed4a58848a10', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:37,095 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:37,095 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.035) securityscanningnotificationworker stdout | 2023-01-17 05:38:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:38:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:37.537407+00:00 (in 0.280322 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:38:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:37 UTC)" (scheduled at 2023-01-17 05:38:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:38:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:38:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 37, 257498), True, datetime.datetime(2023, 1, 17, 5, 38, 37, 257498), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:38:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:38:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:38:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:38:37,429 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:38:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:38:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:37.256652+00:00 (in 59.718781 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:38:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:37 UTC)" (scheduled at 2023-01-17 05:38:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:38:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:38:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:37 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:38:38,020 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:38:38,023 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:38:38,313 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:38:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:38:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:10.738568+00:00 (in 29.999519 seconds) buildlogsarchiver stdout | 2023-01-17 05:38:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:10 UTC)" (scheduled at 2023-01-17 05:38:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:38:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 38, 40, 739446), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:38:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:38:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:38:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:38:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:38:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:42.941536+00:00 (in 1.792813 seconds) namespacegcworker stdout | 2023-01-17 05:38:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:41 UTC)" (scheduled at 2023-01-17 05:38:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:38:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:38:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 41, 149125), True, datetime.datetime(2023, 1, 17, 5, 38, 41, 149125), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:38:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:38:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:38:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:41 UTC)" executed successfully builder stdout | 2023-01-17 05:38:42,540 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:42,553 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:42,553 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:42,563 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:38:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:38:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:41.148245+00:00 (in 58.206247 seconds) namespacegcworker stdout | 2023-01-17 05:38:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:42 UTC)" (scheduled at 2023-01-17 05:38:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:38:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:38:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:39:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:55.646354+00:00 (in 9.999432 seconds) notificationworker stdout | 2023-01-17 05:38:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:55 UTC)" (scheduled at 2023-01-17 05:38:45.646354+00:00) notificationworker stdout | 2023-01-17 05:38:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 45, 647439), True, datetime.datetime(2023, 1, 17, 5, 38, 45, 647439), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:38:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:38:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:38:49.537472+00:00 (in 2.600035 seconds) securityworker stdout | 2023-01-17 05:38:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:16 UTC)" (scheduled at 2023-01-17 05:38:46.936988+00:00) securityworker stdout | 2023-01-17 05:38:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:38:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:38:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:16.936988+00:00 (in 27.399013 seconds) securityworker stdout | 2023-01-17 05:38:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:19 UTC)" (scheduled at 2023-01-17 05:38:49.537472+00:00) securityworker stdout | 2023-01-17 05:38:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:38:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:38:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:38:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:38:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:38:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:38:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:38:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:38:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:38:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:38:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:38:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:38:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:38:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:38:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:38:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:21.348605+00:00 (in 29.999494 seconds) gcworker stdout | 2023-01-17 05:38:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:21 UTC)" (scheduled at 2023-01-17 05:38:51.348605+00:00) gcworker stdout | 2023-01-17 05:38:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724331349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:38:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:38:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:38:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:38:52,060 [231] [DEBUG] [app] Starting request: urn:request:5400ef59-88c4-4688-8309-d8ea1c8956dd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:52,060 [236] [DEBUG] [app] Starting request: urn:request:b3da56f0-2277-4e4e-89a8-3d91084aa39a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:38:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:38:52,065 [242] [DEBUG] [app] Starting request: urn:request:61decd69-974c-4e69-98bc-000c265821b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:52,065 [243] [DEBUG] [app] Starting request: urn:request:c8f4240e-f482-4080-b1e5-00e047b4b39f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:38:52,065 [242] [DEBUG] [app] Ending request: urn:request:61decd69-974c-4e69-98bc-000c265821b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:61decd69-974c-4e69-98bc-000c265821b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:38:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:38:52,066 [243] [DEBUG] [app] Ending request: urn:request:c8f4240e-f482-4080-b1e5-00e047b4b39f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c8f4240e-f482-4080-b1e5-00e047b4b39f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:38:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:38:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:38:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:38:52,070 [233] [DEBUG] [app] Starting request: urn:request:70b7f42b-815c-4a8c-9751-f02162fc93c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:52,071 [236] [DEBUG] [app] Starting request: urn:request:de6141a4-2bbd-470d-a0bc-14100f20ed7c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:38:52,071 [233] [DEBUG] [app] Ending request: urn:request:70b7f42b-815c-4a8c-9751-f02162fc93c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70b7f42b-815c-4a8c-9751-f02162fc93c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:52,071 [236] [DEBUG] [app] Ending request: urn:request:de6141a4-2bbd-470d-a0bc-14100f20ed7c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de6141a4-2bbd-470d-a0bc-14100f20ed7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:38:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:38:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:38:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:38:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:38:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:38:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:38:52,088 [236] [DEBUG] [app] Ending request: urn:request:b3da56f0-2277-4e4e-89a8-3d91084aa39a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b3da56f0-2277-4e4e-89a8-3d91084aa39a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:52,088 [231] [DEBUG] [app] Ending request: urn:request:5400ef59-88c4-4688-8309-d8ea1c8956dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5400ef59-88c4-4688-8309-d8ea1c8956dd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:38:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:38:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:38:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:38:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:38:52,567 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:38:52,579 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:38:52,580 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:38:52,590 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:38:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:01.636362+00:00 (in 5.989368 seconds) notificationworker stdout | 2023-01-17 05:38:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:05 UTC)" (scheduled at 2023-01-17 05:38:55.646354+00:00) notificationworker stdout | 2023-01-17 05:38:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:38:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 38, 55, 647452), True, datetime.datetime(2023, 1, 17, 5, 38, 55, 647452), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:38:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:38:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:38:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:38:59,248 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:38:59,321 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:39:00,189 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:39:00,709 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:39:00,721 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:39:00,997 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:39:01,163 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:39:01,166 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:39:01,168 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:39:01,170 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:39:01,173 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:39:01,254 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:39:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:05.646354+00:00 (in 4.009485 seconds) notificationworker stdout | 2023-01-17 05:39:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:01 UTC)" (scheduled at 2023-01-17 05:39:01.636362+00:00) notificationworker stdout | 2023-01-17 05:39:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:39:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:01 UTC)" executed successfully builder stdout | 2023-01-17 05:39:02,600 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:02,613 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:02,613 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:02,624 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:39:04,420 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:39:04,527 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:39:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:39:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:19.346120+00:00 (in 14.408866 seconds) repositorygcworker stdout | 2023-01-17 05:39:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:04 UTC)" (scheduled at 2023-01-17 05:39:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:39:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:39:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 4, 937689), True, datetime.datetime(2023, 1, 17, 5, 39, 4, 937689), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:39:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:39:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:39:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:39:05,152 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:39:05,155 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:39:05,157 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:39:05,503 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:39:05,505 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:39:05,508 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:39:05,510 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:39:05,512 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:39:05,515 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:39:05,517 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:39:05,520 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:39:05,523 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:39:05,600 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:39:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:15.646354+00:00 (in 9.999460 seconds) notificationworker stdout | 2023-01-17 05:39:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:15 UTC)" (scheduled at 2023-01-17 05:39:05.646354+00:00) notificationworker stdout | 2023-01-17 05:39:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 5, 647395), True, datetime.datetime(2023, 1, 17, 5, 39, 5, 647395), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:15 UTC)" executed successfully servicekey stdout | 2023-01-17 05:39:05,662 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:39:05,856 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:39:06,419 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:39:06,604 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:39:06,856 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:39:06,897 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:39:07,059 [233] [DEBUG] [app] Starting request: urn:request:e5be43b6-3a5e-4375-a4b5-582b88b9047b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:07,060 [231] [DEBUG] [app] Starting request: urn:request:a9e400a6-a5c3-4ad2-bede-7296c8a08324 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:07,064 [242] [DEBUG] [app] Starting request: urn:request:be97c841-7010-48fb-b878-36d44d4d0637 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:07,065 [243] [DEBUG] [app] Starting request: urn:request:e5e71b6c-bf18-42f1-9188-cc9ea5657615 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:07,065 [242] [DEBUG] [app] Ending request: urn:request:be97c841-7010-48fb-b878-36d44d4d0637 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be97c841-7010-48fb-b878-36d44d4d0637', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:39:07,065 [243] [DEBUG] [app] Ending request: urn:request:e5e71b6c-bf18-42f1-9188-cc9ea5657615 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5e71b6c-bf18-42f1-9188-cc9ea5657615', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:39:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:39:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:07,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:39:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:07,070 [236] [DEBUG] [app] Starting request: urn:request:f818453b-53a5-4c97-9fa4-6ac7680fb16c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:07,070 [231] [DEBUG] [app] Starting request: urn:request:7c5e7fda-c327-45b5-8586-7078b1097789 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:07,070 [236] [DEBUG] [app] Ending request: urn:request:f818453b-53a5-4c97-9fa4-6ac7680fb16c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f818453b-53a5-4c97-9fa4-6ac7680fb16c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:07,070 [231] [DEBUG] [app] Ending request: urn:request:7c5e7fda-c327-45b5-8586-7078b1097789 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c5e7fda-c327-45b5-8586-7078b1097789', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:39:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:07,088 [231] [DEBUG] [app] Ending request: urn:request:a9e400a6-a5c3-4ad2-bede-7296c8a08324 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a9e400a6-a5c3-4ad2-bede-7296c8a08324', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:07,088 [233] [DEBUG] [app] Ending request: urn:request:e5be43b6-3a5e-4375-a4b5-582b88b9047b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5be43b6-3a5e-4375-a4b5-582b88b9047b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:39:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) globalpromstats stdout | 2023-01-17 05:39:07,443 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:39:08,030 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:39:08,055 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:39:08,345 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:39:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:39:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:40.738568+00:00 (in 29.999536 seconds) buildlogsarchiver stdout | 2023-01-17 05:39:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:40 UTC)" (scheduled at 2023-01-17 05:39:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:39:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 39, 10, 739412), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:39:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:39:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:39:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:40 UTC)" executed successfully builder stdout | 2023-01-17 05:39:12,625 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:12,637 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:12,638 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:12,648 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:25.646354+00:00 (in 9.999431 seconds) notificationworker stdout | 2023-01-17 05:39:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:25 UTC)" (scheduled at 2023-01-17 05:39:15.646354+00:00) notificationworker stdout | 2023-01-17 05:39:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 15, 647436), True, datetime.datetime(2023, 1, 17, 5, 39, 15, 647436), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:39:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:19.537472+00:00 (in 2.599995 seconds) securityworker stdout | 2023-01-17 05:39:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:46 UTC)" (scheduled at 2023-01-17 05:39:16.936988+00:00) securityworker stdout | 2023-01-17 05:39:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:39:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:39:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:04.936835+00:00 (in 45.590252 seconds) repositorygcworker stdout | 2023-01-17 05:39:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:19 UTC)" (scheduled at 2023-01-17 05:39:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:39:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:39:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:39:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:46.936988+00:00 (in 27.399060 seconds) securityworker stdout | 2023-01-17 05:39:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:49 UTC)" (scheduled at 2023-01-17 05:39:19.537472+00:00) securityworker stdout | 2023-01-17 05:39:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:39:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:39:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:39:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:39:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:39:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:39:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:39:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:39:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:51.348605+00:00 (in 29.999513 seconds) gcworker stdout | 2023-01-17 05:39:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:51 UTC)" (scheduled at 2023-01-17 05:39:21.348605+00:00) gcworker stdout | 2023-01-17 05:39:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724361349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:39:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:39:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:39:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:39:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:39:22,060 [231] [DEBUG] [app] Starting request: urn:request:7fd5ccbd-2368-447f-a518-16e12c629f3c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:22,060 [235] [DEBUG] [app] Starting request: urn:request:6e2946b0-bdf7-4f75-9bd2-6fa9862d5c17 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:22,065 [242] [DEBUG] [app] Starting request: urn:request:470d7319-4f1e-4673-8ab9-590802cf5742 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:22,065 [243] [DEBUG] [app] Starting request: urn:request:98b0eba4-f0f1-430f-bb84-0009a93ac74c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:22,065 [242] [DEBUG] [app] Ending request: urn:request:470d7319-4f1e-4673-8ab9-590802cf5742 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:470d7319-4f1e-4673-8ab9-590802cf5742', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:39:22,065 [243] [DEBUG] [app] Ending request: urn:request:98b0eba4-f0f1-430f-bb84-0009a93ac74c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:98b0eba4-f0f1-430f-bb84-0009a93ac74c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:39:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:39:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:39:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:22,069 [231] [DEBUG] [app] Starting request: urn:request:60d1b0f8-78e9-4272-83f9-2d1ee60fbc0c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:22,070 [231] [DEBUG] [app] Ending request: urn:request:60d1b0f8-78e9-4272-83f9-2d1ee60fbc0c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:60d1b0f8-78e9-4272-83f9-2d1ee60fbc0c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:22,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:39:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:22,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:22,071 [236] [DEBUG] [app] Starting request: urn:request:f28a77ae-d0b8-4086-a074-5ba24c887b74 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:22,072 [236] [DEBUG] [app] Ending request: urn:request:f28a77ae-d0b8-4086-a074-5ba24c887b74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f28a77ae-d0b8-4086-a074-5ba24c887b74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:39:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:22,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:22,086 [235] [DEBUG] [app] Ending request: urn:request:6e2946b0-bdf7-4f75-9bd2-6fa9862d5c17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e2946b0-bdf7-4f75-9bd2-6fa9862d5c17', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:22,086 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:22,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.027) gunicorn-web stdout | 2023-01-17 05:39:22,088 [231] [DEBUG] [app] Ending request: urn:request:7fd5ccbd-2368-447f-a518-16e12c629f3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7fd5ccbd-2368-447f-a518-16e12c629f3c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:39:22,655 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:22,667 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:22,667 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:22,677 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:35.646354+00:00 (in 9.999477 seconds) notificationworker stdout | 2023-01-17 05:39:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:35 UTC)" (scheduled at 2023-01-17 05:39:25.646354+00:00) notificationworker stdout | 2023-01-17 05:39:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 25, 647389), True, datetime.datetime(2023, 1, 17, 5, 39, 25, 647389), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:39:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:39:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:35.853396+00:00 (in 6.889260 seconds) exportactionlogsworker stdout | 2023-01-17 05:39:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:28 UTC)" (scheduled at 2023-01-17 05:39:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:39:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:39:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:29,283 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:39:29,356 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:39:30,206 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:39:30,730 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:39:30,753 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:39:31,032 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:39:31,171 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:39:31,173 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:39:31,176 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:39:31,178 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:39:31,181 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:39:31,289 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:39:32,681 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:32,694 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:32,694 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:32,704 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:39:34,426 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:39:34,559 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:39:35,158 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:39:35,162 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:39:35,164 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:39:35,511 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:39:35,513 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:39:35,516 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:39:35,518 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:39:35,521 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:39:35,525 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:39:35,528 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:39:35,530 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:39:35,533 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:39:35,634 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:39:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:45.646354+00:00 (in 9.999456 seconds) notificationworker stdout | 2023-01-17 05:39:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:45 UTC)" (scheduled at 2023-01-17 05:39:35.646354+00:00) notificationworker stdout | 2023-01-17 05:39:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 35, 647421), True, datetime.datetime(2023, 1, 17, 5, 39, 35, 647421), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:45 UTC)" executed successfully servicekey stdout | 2023-01-17 05:39:35,685 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} exportactionlogsworker stdout | 2023-01-17 05:39:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:39:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:28.963662+00:00 (in 53.109754 seconds) exportactionlogsworker stdout | 2023-01-17 05:39:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:35 UTC)" (scheduled at 2023-01-17 05:39:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:39:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:39:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 35, 854355), True, datetime.datetime(2023, 1, 17, 5, 39, 35, 854355), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:39:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:39:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:39:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:35 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:39:35,891 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:39:36,443 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:39:36,634 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:39:36,886 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:39:36,917 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:39:37,060 [231] [DEBUG] [app] Starting request: urn:request:056a63cb-13e9-46ef-a013-ad837bed51c8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:37,060 [236] [DEBUG] [app] Starting request: urn:request:a6e1eb0f-9860-4d46-97b3-afae7c682825 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:37,064 [243] [DEBUG] [app] Starting request: urn:request:b4a81999-a641-4aed-8551-2664feb849cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:37,065 [243] [DEBUG] [app] Ending request: urn:request:b4a81999-a641-4aed-8551-2664feb849cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b4a81999-a641-4aed-8551-2664feb849cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:39:37,065 [242] [DEBUG] [app] Starting request: urn:request:c2440562-490e-4ae1-a9ae-c2e5b2731d34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:39:37,065 [242] [DEBUG] [app] Ending request: urn:request:c2440562-490e-4ae1-a9ae-c2e5b2731d34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2440562-490e-4ae1-a9ae-c2e5b2731d34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:39:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:39:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:37,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:37,070 [231] [DEBUG] [app] Starting request: urn:request:be4e03ef-5145-406e-b3d1-35458a764232 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:37,070 [231] [DEBUG] [app] Ending request: urn:request:be4e03ef-5145-406e-b3d1-35458a764232 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be4e03ef-5145-406e-b3d1-35458a764232', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:39:37,071 [235] [DEBUG] [app] Starting request: urn:request:bb97aa8c-1d1b-4c23-a665-71d81c669b56 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:37,071 [235] [DEBUG] [app] Ending request: urn:request:bb97aa8c-1d1b-4c23-a665-71d81c669b56 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb97aa8c-1d1b-4c23-a665-71d81c669b56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:39:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:37,088 [231] [DEBUG] [app] Ending request: urn:request:056a63cb-13e9-46ef-a013-ad837bed51c8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:056a63cb-13e9-46ef-a013-ad837bed51c8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:37,088 [236] [DEBUG] [app] Ending request: urn:request:a6e1eb0f-9860-4d46-97b3-afae7c682825 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6e1eb0f-9860-4d46-97b3-afae7c682825', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:39:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:39:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:39:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:39:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:37.537407+00:00 (in 0.280317 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:39:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:37 UTC)" (scheduled at 2023-01-17 05:39:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:39:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:39:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 37, 257507), True, datetime.datetime(2023, 1, 17, 5, 39, 37, 257507), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:39:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:39:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:39:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:39:37,456 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:39:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:39:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:37.256652+00:00 (in 59.718782 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:39:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:37 UTC)" (scheduled at 2023-01-17 05:39:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:39:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:39:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:39:38,060 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:39:38,064 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:39:38,362 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:39:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:39:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:10.738568+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 05:39:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:10 UTC)" (scheduled at 2023-01-17 05:39:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:39:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 39, 40, 739421), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:39:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:39:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:39:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:39:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:39:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:42.941536+00:00 (in 1.792850 seconds) namespacegcworker stdout | 2023-01-17 05:39:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:41 UTC)" (scheduled at 2023-01-17 05:39:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:39:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:39:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 41, 149109), True, datetime.datetime(2023, 1, 17, 5, 39, 41, 149109), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:39:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:39:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:39:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:41 UTC)" executed successfully builder stdout | 2023-01-17 05:39:42,714 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:42,727 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:42,727 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:42,738 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:39:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:39:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:41.148245+00:00 (in 58.206229 seconds) namespacegcworker stdout | 2023-01-17 05:39:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:42 UTC)" (scheduled at 2023-01-17 05:39:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:39:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:39:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:40:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:55.646354+00:00 (in 9.999487 seconds) notificationworker stdout | 2023-01-17 05:39:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:55 UTC)" (scheduled at 2023-01-17 05:39:45.646354+00:00) notificationworker stdout | 2023-01-17 05:39:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 45, 647242), True, datetime.datetime(2023, 1, 17, 5, 39, 45, 647242), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:45,665 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:45,665 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:45,665 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:39:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:39:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:39:49.537472+00:00 (in 2.600033 seconds) securityworker stdout | 2023-01-17 05:39:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:16 UTC)" (scheduled at 2023-01-17 05:39:46.936988+00:00) securityworker stdout | 2023-01-17 05:39:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:39:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:39:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:16.936988+00:00 (in 27.399021 seconds) securityworker stdout | 2023-01-17 05:39:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:19 UTC)" (scheduled at 2023-01-17 05:39:49.537472+00:00) securityworker stdout | 2023-01-17 05:39:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:39:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:39:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:39:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:39:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:39:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:39:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:39:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:39:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:39:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:39:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:39:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:39:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:39:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:39:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:39:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:21.348605+00:00 (in 29.999556 seconds) gcworker stdout | 2023-01-17 05:39:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:21 UTC)" (scheduled at 2023-01-17 05:39:51.348605+00:00) gcworker stdout | 2023-01-17 05:39:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724391349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:39:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:39:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:39:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:39:52,060 [231] [DEBUG] [app] Starting request: urn:request:18d6a537-26a5-4d17-a5a3-4cd5a8b6712a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:52,060 [236] [DEBUG] [app] Starting request: urn:request:fe206e10-2474-400a-b5e2-677af56752b3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:39:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:39:52,065 [243] [DEBUG] [app] Starting request: urn:request:c5a4df5b-f475-4a1c-902d-0466fd75a1d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:52,065 [230] [DEBUG] [app] Starting request: urn:request:f2907d42-d7c0-4d00-ba9a-c542889a75b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:39:52,065 [243] [DEBUG] [app] Ending request: urn:request:c5a4df5b-f475-4a1c-902d-0466fd75a1d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5a4df5b-f475-4a1c-902d-0466fd75a1d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:39:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:39:52,066 [230] [DEBUG] [app] Ending request: urn:request:f2907d42-d7c0-4d00-ba9a-c542889a75b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f2907d42-d7c0-4d00-ba9a-c542889a75b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:39:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:39:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:39:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:39:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:52,069 [236] [DEBUG] [app] Starting request: urn:request:06762640-4e53-4c97-b196-be444fdd0b9e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:52,070 [236] [DEBUG] [app] Ending request: urn:request:06762640-4e53-4c97-b196-be444fdd0b9e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06762640-4e53-4c97-b196-be444fdd0b9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:52,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:39:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:52,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:39:52,071 [235] [DEBUG] [app] Starting request: urn:request:cd5843bd-fef0-4185-8317-f27e73322c8f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:39:52,072 [235] [DEBUG] [app] Ending request: urn:request:cd5843bd-fef0-4185-8317-f27e73322c8f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cd5843bd-fef0-4185-8317-f27e73322c8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:39:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:39:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:39:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:39:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:39:52,086 [231] [DEBUG] [app] Ending request: urn:request:18d6a537-26a5-4d17-a5a3-4cd5a8b6712a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:18d6a537-26a5-4d17-a5a3-4cd5a8b6712a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:52,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:52,086 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.027) gunicorn-web stdout | 2023-01-17 05:39:52,088 [236] [DEBUG] [app] Ending request: urn:request:fe206e10-2474-400a-b5e2-677af56752b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fe206e10-2474-400a-b5e2-677af56752b3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:39:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:39:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:39:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:39:52,742 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:39:52,758 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:39:52,758 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:39:52,768 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:39:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:01.636362+00:00 (in 5.989546 seconds) notificationworker stdout | 2023-01-17 05:39:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:05 UTC)" (scheduled at 2023-01-17 05:39:55.646354+00:00) notificationworker stdout | 2023-01-17 05:39:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:39:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 39, 55, 647239), True, datetime.datetime(2023, 1, 17, 5, 39, 55, 647239), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:39:55,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:39:55,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:39:55,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:39:59,309 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:39:59,389 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:40:00,230 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:40:00,757 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:40:00,788 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:40:01,063 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:40:01,178 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:40:01,180 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:40:01,183 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:40:01,185 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:40:01,187 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:40:01,311 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:40:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:05.646354+00:00 (in 4.009492 seconds) notificationworker stdout | 2023-01-17 05:40:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:01 UTC)" (scheduled at 2023-01-17 05:40:01.636362+00:00) notificationworker stdout | 2023-01-17 05:40:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:40:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:01 UTC)" executed successfully builder stdout | 2023-01-17 05:40:02,778 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:02,791 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:02,791 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:02,802 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:40:04,462 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:40:04,590 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:40:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:40:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:19.346120+00:00 (in 14.408782 seconds) repositorygcworker stdout | 2023-01-17 05:40:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:04 UTC)" (scheduled at 2023-01-17 05:40:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:40:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:40:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 4, 937797), True, datetime.datetime(2023, 1, 17, 5, 40, 4, 937797), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:40:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:40:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:40:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:40:05,166 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:40:05,168 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:40:05,170 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:40:05,518 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:40:05,521 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:40:05,524 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:40:05,526 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:40:05,529 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:40:05,534 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:40:05,536 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:40:05,538 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:40:05,543 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:40:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:15.646354+00:00 (in 9.999365 seconds) notificationworker stdout | 2023-01-17 05:40:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:15 UTC)" (scheduled at 2023-01-17 05:40:05.646354+00:00) notificationworker stdout | 2023-01-17 05:40:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 5, 647435), True, datetime.datetime(2023, 1, 17, 5, 40, 5, 647435), 0, 'notification/%', 50, 1, 0]) storagereplication stdout | 2023-01-17 05:40:05,656 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} notificationworker stdout | 2023-01-17 05:40:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:15 UTC)" executed successfully servicekey stdout | 2023-01-17 05:40:05,718 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:40:05,905 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:40:06,479 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:40:06,649 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:40:06,904 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:40:06,952 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:40:07,061 [231] [DEBUG] [app] Starting request: urn:request:1f2e760c-696d-4245-bbee-1edbb4a5490f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:07,061 [236] [DEBUG] [app] Starting request: urn:request:67769055-474b-4306-8146-b93cf81601e8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:07,066 [243] [DEBUG] [app] Starting request: urn:request:b63b8789-9a6c-4633-9cd9-f3e057e26cd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:07,066 [240] [DEBUG] [app] Starting request: urn:request:5304d986-bdef-4726-adb5-e752365b7819 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:07,066 [243] [DEBUG] [app] Ending request: urn:request:b63b8789-9a6c-4633-9cd9-f3e057e26cd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b63b8789-9a6c-4633-9cd9-f3e057e26cd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:40:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:40:07,066 [240] [DEBUG] [app] Ending request: urn:request:5304d986-bdef-4726-adb5-e752365b7819 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5304d986-bdef-4726-adb5-e752365b7819', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:07,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:40:07,067 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:40:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:07,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:07,071 [235] [DEBUG] [app] Starting request: urn:request:bd24ea53-5763-4271-b7f7-e9f3d0c37f8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:07,072 [233] [DEBUG] [app] Starting request: urn:request:322b4a44-7872-4384-a446-7474d44f48db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:07,072 [235] [DEBUG] [app] Ending request: urn:request:bd24ea53-5763-4271-b7f7-e9f3d0c37f8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bd24ea53-5763-4271-b7f7-e9f3d0c37f8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:07,072 [233] [DEBUG] [app] Ending request: urn:request:322b4a44-7872-4384-a446-7474d44f48db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:322b4a44-7872-4384-a446-7474d44f48db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:40:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:07,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:07,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:07,090 [231] [DEBUG] [app] Ending request: urn:request:1f2e760c-696d-4245-bbee-1edbb4a5490f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1f2e760c-696d-4245-bbee-1edbb4a5490f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:07,090 [236] [DEBUG] [app] Ending request: urn:request:67769055-474b-4306-8146-b93cf81601e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:67769055-474b-4306-8146-b93cf81601e8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:07,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:07,090 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:40:07,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:40:07,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" globalpromstats stdout | 2023-01-17 05:40:07,463 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:40:08,079 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:40:08,100 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:40:08,386 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:40:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:40:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:40.738568+00:00 (in 29.999505 seconds) buildlogsarchiver stdout | 2023-01-17 05:40:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:40 UTC)" (scheduled at 2023-01-17 05:40:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:40:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 40, 10, 739470), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:40:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:40:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:40:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:40 UTC)" executed successfully builder stdout | 2023-01-17 05:40:12,812 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:12,824 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:12,824 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:12,835 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:25.646354+00:00 (in 9.999475 seconds) notificationworker stdout | 2023-01-17 05:40:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:25 UTC)" (scheduled at 2023-01-17 05:40:15.646354+00:00) notificationworker stdout | 2023-01-17 05:40:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 15, 647280), True, datetime.datetime(2023, 1, 17, 5, 40, 15, 647280), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:40:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:19.537472+00:00 (in 2.600023 seconds) securityworker stdout | 2023-01-17 05:40:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:46 UTC)" (scheduled at 2023-01-17 05:40:16.936988+00:00) securityworker stdout | 2023-01-17 05:40:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:40:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:40:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:04.936835+00:00 (in 45.590248 seconds) repositorygcworker stdout | 2023-01-17 05:40:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:19 UTC)" (scheduled at 2023-01-17 05:40:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:40:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:40:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:40:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:46.936988+00:00 (in 27.399044 seconds) securityworker stdout | 2023-01-17 05:40:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:49 UTC)" (scheduled at 2023-01-17 05:40:19.537472+00:00) securityworker stdout | 2023-01-17 05:40:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:40:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:40:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:40:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:40:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:40:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:40:19,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:40:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:40:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:51.348605+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 05:40:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:51 UTC)" (scheduled at 2023-01-17 05:40:21.348605+00:00) gcworker stdout | 2023-01-17 05:40:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724421349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:40:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:40:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:40:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:40:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:40:22,060 [235] [DEBUG] [app] Starting request: urn:request:2dc7611e-0035-4890-a519-037947aca6d6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:22,060 [233] [DEBUG] [app] Starting request: urn:request:82a0ad9f-8a39-44da-aa26-230b5a594b71 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:22,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:22,065 [242] [DEBUG] [app] Starting request: urn:request:eba5f06e-5f77-4d92-92f5-0091eb3a789a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:22,066 [242] [DEBUG] [app] Ending request: urn:request:eba5f06e-5f77-4d92-92f5-0091eb3a789a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eba5f06e-5f77-4d92-92f5-0091eb3a789a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:22,066 [230] [DEBUG] [app] Starting request: urn:request:ce3ff12d-d0b2-42c8-9d8e-50c3a466d1fe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:40:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:40:22,066 [230] [DEBUG] [app] Ending request: urn:request:ce3ff12d-d0b2-42c8-9d8e-50c3a466d1fe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ce3ff12d-d0b2-42c8-9d8e-50c3a466d1fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:22,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:40:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:22,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:22,070 [235] [DEBUG] [app] Starting request: urn:request:b3ccf32e-f3df-4d03-9a62-aed9dc94389f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:22,071 [235] [DEBUG] [app] Ending request: urn:request:b3ccf32e-f3df-4d03-9a62-aed9dc94389f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3ccf32e-f3df-4d03-9a62-aed9dc94389f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:22,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:22,072 [231] [DEBUG] [app] Starting request: urn:request:b93d0ea7-ef24-49c5-955d-342195ef8b6b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:22,073 [231] [DEBUG] [app] Ending request: urn:request:b93d0ea7-ef24-49c5-955d-342195ef8b6b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b93d0ea7-ef24-49c5-955d-342195ef8b6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:40:22,073 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:22,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:22,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:22,087 [233] [DEBUG] [app] Ending request: urn:request:82a0ad9f-8a39-44da-aa26-230b5a594b71 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:82a0ad9f-8a39-44da-aa26-230b5a594b71', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:40:22,089 [235] [DEBUG] [app] Ending request: urn:request:2dc7611e-0035-4890-a519-037947aca6d6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2dc7611e-0035-4890-a519-037947aca6d6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 05:40:22,845 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:22,862 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:22,862 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:22,872 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:35.646354+00:00 (in 9.999510 seconds) notificationworker stdout | 2023-01-17 05:40:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:35 UTC)" (scheduled at 2023-01-17 05:40:25.646354+00:00) notificationworker stdout | 2023-01-17 05:40:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 25, 647247), True, datetime.datetime(2023, 1, 17, 5, 40, 25, 647247), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:40:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:40:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:35.853396+00:00 (in 6.889271 seconds) exportactionlogsworker stdout | 2023-01-17 05:40:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:28 UTC)" (scheduled at 2023-01-17 05:40:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:40:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:40:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:29,340 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:40:29,395 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:40:30,261 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:40:30,793 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:40:30,800 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:40:31,069 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:40:31,184 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:40:31,187 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:40:31,189 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:40:31,191 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:40:31,194 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:40:31,318 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:40:32,873 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:32,885 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:32,885 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:32,896 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:40:34,496 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:40:34,622 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:40:35,173 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:40:35,176 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:40:35,178 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:40:35,531 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:40:35,533 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:40:35,536 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:40:35,538 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:40:35,541 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:40:35,543 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:40:35,546 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:40:35,548 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:40:35,552 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:40:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:45.646354+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 05:40:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:45 UTC)" (scheduled at 2023-01-17 05:40:35.646354+00:00) notificationworker stdout | 2023-01-17 05:40:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 35, 647275), True, datetime.datetime(2023, 1, 17, 5, 40, 35, 647275), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:45 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:40:35,687 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:40:35,738 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} exportactionlogsworker stdout | 2023-01-17 05:40:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:40:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:28.963662+00:00 (in 53.109803 seconds) exportactionlogsworker stdout | 2023-01-17 05:40:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:35 UTC)" (scheduled at 2023-01-17 05:40:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:40:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:40:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 35, 854267), True, datetime.datetime(2023, 1, 17, 5, 40, 35, 854267), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:40:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:40:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:40:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:35 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:40:35,935 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:40:36,505 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:40:36,669 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:40:36,939 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:40:36,958 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:40:37,060 [231] [DEBUG] [app] Starting request: urn:request:a09dcae8-b889-47ff-b93f-8938fffa2769 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:37,060 [235] [DEBUG] [app] Starting request: urn:request:abbd2aea-c35d-4222-a318-0a585b8a8918 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:37,065 [230] [DEBUG] [app] Starting request: urn:request:933da8d9-41df-4734-b472-9cc55f23c516 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:37,065 [230] [DEBUG] [app] Ending request: urn:request:933da8d9-41df-4734-b472-9cc55f23c516 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:933da8d9-41df-4734-b472-9cc55f23c516', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:37,065 [243] [DEBUG] [app] Starting request: urn:request:db3abecc-633b-4d9a-ac30-57343ee71a98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:40:37,066 [243] [DEBUG] [app] Ending request: urn:request:db3abecc-633b-4d9a-ac30-57343ee71a98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:db3abecc-633b-4d9a-ac30-57343ee71a98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:40:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:40:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:37,070 [231] [DEBUG] [app] Starting request: urn:request:bff49360-ebd3-444a-9043-de00139a034b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:37,071 [231] [DEBUG] [app] Ending request: urn:request:bff49360-ebd3-444a-9043-de00139a034b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bff49360-ebd3-444a-9043-de00139a034b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:37,071 [236] [DEBUG] [app] Starting request: urn:request:31eb8ce7-ae71-4469-89a0-f1fc54f39c82 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:40:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:37,071 [236] [DEBUG] [app] Ending request: urn:request:31eb8ce7-ae71-4469-89a0-f1fc54f39c82 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31eb8ce7-ae71-4469-89a0-f1fc54f39c82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:37,088 [231] [DEBUG] [app] Ending request: urn:request:a09dcae8-b889-47ff-b93f-8938fffa2769 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a09dcae8-b889-47ff-b93f-8938fffa2769', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:37,088 [235] [DEBUG] [app] Ending request: urn:request:abbd2aea-c35d-4222-a318-0a585b8a8918 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:abbd2aea-c35d-4222-a318-0a585b8a8918', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:40:37,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:40:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:40:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:40:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:37.537407+00:00 (in 0.280343 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:40:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:37 UTC)" (scheduled at 2023-01-17 05:40:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:40:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:40:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 37, 257484), True, datetime.datetime(2023, 1, 17, 5, 40, 37, 257484), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:40:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:40:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:40:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:40:37,499 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} securityscanningnotificationworker stdout | 2023-01-17 05:40:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:40:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:23.537048+00:00 (in 45.999272 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:40:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:37 UTC)" (scheduled at 2023-01-17 05:40:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:40:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:40:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:37 UTC)" executed successfully teamsyncworker stdout | 2023-01-17 05:40:38,111 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:40:38,135 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:40:38,422 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:40:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:40:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:10.738568+00:00 (in 29.999521 seconds) buildlogsarchiver stdout | 2023-01-17 05:40:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:10 UTC)" (scheduled at 2023-01-17 05:40:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:40:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 40, 40, 739455), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:40:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:40:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:40:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:40:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:40:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:42.941536+00:00 (in 1.792826 seconds) namespacegcworker stdout | 2023-01-17 05:40:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:41 UTC)" (scheduled at 2023-01-17 05:40:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:40:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:40:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 41, 149113), True, datetime.datetime(2023, 1, 17, 5, 40, 41, 149113), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:40:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:40:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:40:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:41 UTC)" executed successfully builder stdout | 2023-01-17 05:40:42,906 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:42,918 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:42,918 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:42,929 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:40:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:40:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:41.148245+00:00 (in 58.206274 seconds) namespacegcworker stdout | 2023-01-17 05:40:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:42 UTC)" (scheduled at 2023-01-17 05:40:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:40:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:40:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:41:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:55.646354+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 05:40:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:55 UTC)" (scheduled at 2023-01-17 05:40:45.646354+00:00) notificationworker stdout | 2023-01-17 05:40:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 45, 647271), True, datetime.datetime(2023, 1, 17, 5, 40, 45, 647271), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:40:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:40:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:40:49.537472+00:00 (in 2.600036 seconds) securityworker stdout | 2023-01-17 05:40:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:16 UTC)" (scheduled at 2023-01-17 05:40:46.936988+00:00) securityworker stdout | 2023-01-17 05:40:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:40:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:40:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:16.936988+00:00 (in 27.399041 seconds) securityworker stdout | 2023-01-17 05:40:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:19 UTC)" (scheduled at 2023-01-17 05:40:49.537472+00:00) securityworker stdout | 2023-01-17 05:40:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:40:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:40:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:40:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:40:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:40:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:40:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:40:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:40:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:40:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:40:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:40:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:40:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:40:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:40:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:40:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:21.348605+00:00 (in 29.999502 seconds) gcworker stdout | 2023-01-17 05:40:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:21 UTC)" (scheduled at 2023-01-17 05:40:51.348605+00:00) gcworker stdout | 2023-01-17 05:40:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724451349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:40:51,364 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:40:51,364 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:40:51,364 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:40:52,060 [231] [DEBUG] [app] Starting request: urn:request:d0deb507-ee36-42b9-bab7-a6a065ba9090 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:52,060 [235] [DEBUG] [app] Starting request: urn:request:f1888a21-4fbd-417c-9008-33d8257240fa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:40:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:52,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:40:52,065 [230] [DEBUG] [app] Starting request: urn:request:6a931a74-0a4e-49dd-8ea9-75d474b9453a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:40:52,065 [230] [DEBUG] [app] Ending request: urn:request:6a931a74-0a4e-49dd-8ea9-75d474b9453a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6a931a74-0a4e-49dd-8ea9-75d474b9453a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:52,065 [243] [DEBUG] [app] Starting request: urn:request:f5d5d7f7-afe4-4a1b-891d-7431d8b580b6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:40:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:40:52,066 [243] [DEBUG] [app] Ending request: urn:request:f5d5d7f7-afe4-4a1b-891d-7431d8b580b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5d5d7f7-afe4-4a1b-891d-7431d8b580b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:40:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:40:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:40:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:52,070 [236] [DEBUG] [app] Starting request: urn:request:6fa8f577-96f3-4eb5-af5f-3e10ea2bf487 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:52,070 [236] [DEBUG] [app] Ending request: urn:request:6fa8f577-96f3-4eb5-af5f-3e10ea2bf487 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6fa8f577-96f3-4eb5-af5f-3e10ea2bf487', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:40:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:40:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:52,072 [233] [DEBUG] [app] Starting request: urn:request:d9a7898a-069c-4799-bc74-7c15b5fb38fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:40:52,072 [233] [DEBUG] [app] Ending request: urn:request:d9a7898a-069c-4799-bc74-7c15b5fb38fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9a7898a-069c-4799-bc74-7c15b5fb38fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:40:52,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:40:52,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:40:52,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:40:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:40:52,087 [231] [DEBUG] [app] Ending request: urn:request:d0deb507-ee36-42b9-bab7-a6a065ba9090 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d0deb507-ee36-42b9-bab7-a6a065ba9090', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:40:52,088 [235] [DEBUG] [app] Ending request: urn:request:f1888a21-4fbd-417c-9008-33d8257240fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1888a21-4fbd-417c-9008-33d8257240fa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:40:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:40:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:40:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) builder stdout | 2023-01-17 05:40:52,939 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:40:52,951 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:40:52,951 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:40:52,962 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:40:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:01.636362+00:00 (in 5.989493 seconds) notificationworker stdout | 2023-01-17 05:40:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:05 UTC)" (scheduled at 2023-01-17 05:40:55.646354+00:00) notificationworker stdout | 2023-01-17 05:40:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:40:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 40, 55, 647341), True, datetime.datetime(2023, 1, 17, 5, 40, 55, 647341), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:40:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:40:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:40:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:40:59,376 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:40:59,402 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:41:00,287 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:41:00,829 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:41:00,831 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:41:01,104 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:41:01,192 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:41:01,194 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:41:01,196 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:41:01,199 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:41:01,202 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:41:01,353 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:41:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:05.646354+00:00 (in 4.009503 seconds) notificationworker stdout | 2023-01-17 05:41:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:01 UTC)" (scheduled at 2023-01-17 05:41:01.636362+00:00) notificationworker stdout | 2023-01-17 05:41:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:41:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:01 UTC)" executed successfully builder stdout | 2023-01-17 05:41:02,972 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:02,986 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:02,986 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:02,996 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:41:04,508 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:41:04,631 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:41:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:41:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:17.345601+00:00 (in 12.408326 seconds) repositorygcworker stdout | 2023-01-17 05:41:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:04 UTC)" (scheduled at 2023-01-17 05:41:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:41:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:41:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 4, 937705), True, datetime.datetime(2023, 1, 17, 5, 41, 4, 937705), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:41:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:41:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:41:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:41:05,179 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:41:05,182 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:41:05,184 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:41:05,541 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:41:05,543 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:41:05,546 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:41:05,548 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:41:05,550 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:41:05,552 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:41:05,555 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:41:05,558 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:41:05,561 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:41:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:15.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 05:41:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:15 UTC)" (scheduled at 2023-01-17 05:41:05.646354+00:00) notificationworker stdout | 2023-01-17 05:41:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 5, 647195), True, datetime.datetime(2023, 1, 17, 5, 41, 5, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:15 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:41:05,720 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:41:05,747 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:41:05,970 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:41:06,517 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:41:06,704 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:41:06,951 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:41:06,963 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:41:07,059 [235] [DEBUG] [app] Starting request: urn:request:174172a2-3c8b-4f20-a193-2683a136121e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:07,060 [236] [DEBUG] [app] Starting request: urn:request:7ed70678-635a-4042-833b-c91184b2e9ce (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:07,064 [239] [DEBUG] [app] Starting request: urn:request:694856bc-426d-4d1d-92ed-d875f0574188 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:07,065 [240] [DEBUG] [app] Starting request: urn:request:c439ccb3-8742-45dd-93f5-5c61225a0527 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:07,065 [239] [DEBUG] [app] Ending request: urn:request:694856bc-426d-4d1d-92ed-d875f0574188 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:694856bc-426d-4d1d-92ed-d875f0574188', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:41:07,065 [240] [DEBUG] [app] Ending request: urn:request:c439ccb3-8742-45dd-93f5-5c61225a0527 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c439ccb3-8742-45dd-93f5-5c61225a0527', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:41:07,065 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:41:07,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:41:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:07,069 [233] [DEBUG] [app] Starting request: urn:request:3e8bd785-5e83-4f66-86dd-64af1afc02d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:07,070 [233] [DEBUG] [app] Ending request: urn:request:3e8bd785-5e83-4f66-86dd-64af1afc02d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e8bd785-5e83-4f66-86dd-64af1afc02d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:07,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:41:07,071 [231] [DEBUG] [app] Starting request: urn:request:7248f7de-9428-4faa-8a15-7367897ac71a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:07,071 [231] [DEBUG] [app] Ending request: urn:request:7248f7de-9428-4faa-8a15-7367897ac71a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7248f7de-9428-4faa-8a15-7367897ac71a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:07,087 [236] [DEBUG] [app] Ending request: urn:request:7ed70678-635a-4042-833b-c91184b2e9ce (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7ed70678-635a-4042-833b-c91184b2e9ce', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:41:07,088 [235] [DEBUG] [app] Ending request: urn:request:174172a2-3c8b-4f20-a193-2683a136121e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:174172a2-3c8b-4f20-a193-2683a136121e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) globalpromstats stdout | 2023-01-17 05:41:07,535 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:41:08,126 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:41:08,147 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:41:08,457 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:41:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:41:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:40.738568+00:00 (in 29.999537 seconds) buildlogsarchiver stdout | 2023-01-17 05:41:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:40 UTC)" (scheduled at 2023-01-17 05:41:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:41:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 41, 10, 739398), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:41:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:41:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:41:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:40 UTC)" executed successfully builder stdout | 2023-01-17 05:41:13,007 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:13,019 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:13,019 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:13,030 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:25.646354+00:00 (in 9.999568 seconds) notificationworker stdout | 2023-01-17 05:41:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:25 UTC)" (scheduled at 2023-01-17 05:41:15.646354+00:00) notificationworker stdout | 2023-01-17 05:41:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 15, 647221), True, datetime.datetime(2023, 1, 17, 5, 41, 15, 647221), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:41:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:19.537472+00:00 (in 2.600040 seconds) securityworker stdout | 2023-01-17 05:41:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:46 UTC)" (scheduled at 2023-01-17 05:41:16.936988+00:00) securityworker stdout | 2023-01-17 05:41:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:41:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:41:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:19.346120+00:00 (in 2.000050 seconds) repositorygcworker stdout | 2023-01-17 05:41:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:17 UTC)" (scheduled at 2023-01-17 05:41:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:41:17,346 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 41, 17, 346459), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:41:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 17, 346459), True, datetime.datetime(2023, 1, 17, 5, 41, 17, 346459), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:41:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 17, 346459), True, datetime.datetime(2023, 1, 17, 5, 41, 17, 346459), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 41, 17, 346459), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:41:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:41:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:41:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:41:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:04.936835+00:00 (in 45.590248 seconds) repositorygcworker stdout | 2023-01-17 05:41:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:19 UTC)" (scheduled at 2023-01-17 05:41:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:41:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:41:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:41:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:46.936988+00:00 (in 27.399065 seconds) securityworker stdout | 2023-01-17 05:41:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:49 UTC)" (scheduled at 2023-01-17 05:41:19.537472+00:00) securityworker stdout | 2023-01-17 05:41:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:41:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:41:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:19,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:19,540 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:41:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:41:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:41:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:41:19,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:41:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:41:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:51.348605+00:00 (in 29.999492 seconds) gcworker stdout | 2023-01-17 05:41:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:51 UTC)" (scheduled at 2023-01-17 05:41:21.348605+00:00) gcworker stdout | 2023-01-17 05:41:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724481349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:41:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:41:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:41:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:41:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:41:22,060 [235] [DEBUG] [app] Starting request: urn:request:094bfaaa-3959-434c-934c-acc3627d4702 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:22,060 [236] [DEBUG] [app] Starting request: urn:request:b2f71049-3d9d-40b5-868b-2552fbadfe9c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:22,065 [243] [DEBUG] [app] Starting request: urn:request:59d35b6d-5d82-44d8-b219-9f68d06cc7ed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:22,065 [241] [DEBUG] [app] Starting request: urn:request:41c76762-48a8-4224-a0d0-bcfa30fd66c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:22,065 [243] [DEBUG] [app] Ending request: urn:request:59d35b6d-5d82-44d8-b219-9f68d06cc7ed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59d35b6d-5d82-44d8-b219-9f68d06cc7ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:41:22,066 [241] [DEBUG] [app] Ending request: urn:request:41c76762-48a8-4224-a0d0-bcfa30fd66c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:41c76762-48a8-4224-a0d0-bcfa30fd66c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:41:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:41:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.003) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:41:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:22,071 [231] [DEBUG] [app] Starting request: urn:request:17f11b1a-5a74-48c3-9f3f-c59ced948545 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:22,071 [233] [DEBUG] [app] Starting request: urn:request:0b8a4e2a-4a95-48b4-9c8d-f423006cfe5d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:22,071 [231] [DEBUG] [app] Ending request: urn:request:17f11b1a-5a74-48c3-9f3f-c59ced948545 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17f11b1a-5a74-48c3-9f3f-c59ced948545', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:41:22,071 [233] [DEBUG] [app] Ending request: urn:request:0b8a4e2a-4a95-48b4-9c8d-f423006cfe5d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b8a4e2a-4a95-48b4-9c8d-f423006cfe5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:22,088 [236] [DEBUG] [app] Ending request: urn:request:b2f71049-3d9d-40b5-868b-2552fbadfe9c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2f71049-3d9d-40b5-868b-2552fbadfe9c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:22,088 [235] [DEBUG] [app] Ending request: urn:request:094bfaaa-3959-434c-934c-acc3627d4702 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:094bfaaa-3959-434c-934c-acc3627d4702', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:41:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 05:41:23,040 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:23,052 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:23,052 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:23,063 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:41:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:41:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:37.256652+00:00 (in 13.719102 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:41:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:23 UTC)" (scheduled at 2023-01-17 05:41:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:41:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 41, 23, 537945), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:41:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 23, 537945), True, datetime.datetime(2023, 1, 17, 5, 41, 23, 537945), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:41:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 23, 537945), True, datetime.datetime(2023, 1, 17, 5, 41, 23, 537945), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 41, 23, 537945), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:41:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:41:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:35.646354+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 05:41:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:35 UTC)" (scheduled at 2023-01-17 05:41:25.646354+00:00) notificationworker stdout | 2023-01-17 05:41:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 25, 647276), True, datetime.datetime(2023, 1, 17, 5, 41, 25, 647276), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:41:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:41:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:35.853396+00:00 (in 6.889294 seconds) exportactionlogsworker stdout | 2023-01-17 05:41:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:28 UTC)" (scheduled at 2023-01-17 05:41:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:41:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:41:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:29,412 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:41:29,438 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:41:30,322 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:41:30,855 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:41:30,864 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:41:31,140 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:41:31,198 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:41:31,200 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:41:31,204 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:41:31,206 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:41:31,208 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:41:31,388 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:41:33,073 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:33,086 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:33,086 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:33,096 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:41:34,544 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:41:34,652 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:41:35,186 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:41:35,188 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:41:35,191 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:41:35,552 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:41:35,555 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:41:35,557 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:41:35,560 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:41:35,562 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:41:35,565 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:41:35,567 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:41:35,569 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:41:35,572 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:41:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:45.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:41:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:45 UTC)" (scheduled at 2023-01-17 05:41:35.646354+00:00) notificationworker stdout | 2023-01-17 05:41:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 35, 647209), True, datetime.datetime(2023, 1, 17, 5, 41, 35, 647209), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:45 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:41:35,754 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:41:35,782 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} exportactionlogsworker stdout | 2023-01-17 05:41:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:41:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:54.963413+00:00 (in 19.109484 seconds) exportactionlogsworker stdout | 2023-01-17 05:41:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:35 UTC)" (scheduled at 2023-01-17 05:41:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:41:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:41:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 35, 854373), True, datetime.datetime(2023, 1, 17, 5, 41, 35, 854373), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:41:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:41:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:41:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:35 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:41:36,005 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:41:36,524 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:41:36,711 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:41:36,965 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:41:36,998 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:41:37,060 [233] [DEBUG] [app] Starting request: urn:request:85af8c4a-7774-4d3e-839b-023cf877fe0f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:37,060 [236] [DEBUG] [app] Starting request: urn:request:42986398-9693-4de9-8f9a-374358b6c3f6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:37,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:37,065 [243] [DEBUG] [app] Starting request: urn:request:ed391627-9f9b-4d1c-924d-1a4a6f58da57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:37,065 [240] [DEBUG] [app] Starting request: urn:request:df711d81-ccd5-44fc-ad9d-908ea2e5ecbf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:37,066 [243] [DEBUG] [app] Ending request: urn:request:ed391627-9f9b-4d1c-924d-1a4a6f58da57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ed391627-9f9b-4d1c-924d-1a4a6f58da57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:41:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:41:37,066 [240] [DEBUG] [app] Ending request: urn:request:df711d81-ccd5-44fc-ad9d-908ea2e5ecbf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df711d81-ccd5-44fc-ad9d-908ea2e5ecbf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:41:37,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:41:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:37,071 [231] [DEBUG] [app] Starting request: urn:request:8b47f2b4-9198-42c5-bd56-9d45e229ea08 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:37,071 [231] [DEBUG] [app] Ending request: urn:request:8b47f2b4-9198-42c5-bd56-9d45e229ea08 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b47f2b4-9198-42c5-bd56-9d45e229ea08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:37,071 [233] [DEBUG] [app] Starting request: urn:request:110a385e-634f-4483-afe9-61166090f6db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:37,072 [233] [DEBUG] [app] Ending request: urn:request:110a385e-634f-4483-afe9-61166090f6db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:110a385e-634f-4483-afe9-61166090f6db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:41:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:37,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:37,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:37,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:37,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:37,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:37,090 [233] [DEBUG] [app] Ending request: urn:request:85af8c4a-7774-4d3e-839b-023cf877fe0f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:85af8c4a-7774-4d3e-839b-023cf877fe0f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:37,090 [236] [DEBUG] [app] Ending request: urn:request:42986398-9693-4de9-8f9a-374358b6c3f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42986398-9693-4de9-8f9a-374358b6c3f6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:37,090 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:37,090 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:37,090 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:41:37,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) securityscanningnotificationworker stdout | 2023-01-17 05:41:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:41:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:37.537407+00:00 (in 0.280326 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:41:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:37 UTC)" (scheduled at 2023-01-17 05:41:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:41:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:41:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 37, 257451), True, datetime.datetime(2023, 1, 17, 5, 41, 37, 257451), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:41:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:41:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:41:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:41:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:41:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:37.256652+00:00 (in 59.718843 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:41:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:37 UTC)" (scheduled at 2023-01-17 05:41:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:41:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:41:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:41:37,549 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:41:38,154 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:41:38,160 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:41:38,493 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:41:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:41:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:10.738568+00:00 (in 29.999543 seconds) buildlogsarchiver stdout | 2023-01-17 05:41:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:10 UTC)" (scheduled at 2023-01-17 05:41:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:41:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 41, 40, 739402), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:41:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:41:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:41:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:41:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:41:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:42.941536+00:00 (in 1.792840 seconds) namespacegcworker stdout | 2023-01-17 05:41:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:41 UTC)" (scheduled at 2023-01-17 05:41:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:41:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:41:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 41, 149146), True, datetime.datetime(2023, 1, 17, 5, 41, 41, 149146), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:41:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:41:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:41:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:41:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:41:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:06.941185+00:00 (in 23.999138 seconds) namespacegcworker stdout | 2023-01-17 05:41:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:42 UTC)" (scheduled at 2023-01-17 05:41:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:41:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:41:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:42:42 UTC)" executed successfully builder stdout | 2023-01-17 05:41:43,106 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:43,119 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:43,119 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:43,130 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:55.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 05:41:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:55 UTC)" (scheduled at 2023-01-17 05:41:45.646354+00:00) notificationworker stdout | 2023-01-17 05:41:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 45, 647259), True, datetime.datetime(2023, 1, 17, 5, 41, 45, 647259), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:41:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:41:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:41:49.537472+00:00 (in 2.600036 seconds) securityworker stdout | 2023-01-17 05:41:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:16 UTC)" (scheduled at 2023-01-17 05:41:46.936988+00:00) securityworker stdout | 2023-01-17 05:41:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:41:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:41:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:16.936988+00:00 (in 27.399046 seconds) securityworker stdout | 2023-01-17 05:41:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:19 UTC)" (scheduled at 2023-01-17 05:41:49.537472+00:00) securityworker stdout | 2023-01-17 05:41:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:41:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:41:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:41:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:41:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:41:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:41:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:41:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:41:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:41:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:41:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:41:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:41:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:41:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:41:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:41:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:21.348605+00:00 (in 29.999496 seconds) gcworker stdout | 2023-01-17 05:41:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:21 UTC)" (scheduled at 2023-01-17 05:41:51.348605+00:00) gcworker stdout | 2023-01-17 05:41:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724511349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:41:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:41:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:41:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:41:52,061 [231] [DEBUG] [app] Starting request: urn:request:1cf1773b-3f6f-4a7c-b895-ff6fe6ba0361 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:52,061 [236] [DEBUG] [app] Starting request: urn:request:9cc6bb52-5d57-433c-85fb-a94f5fd8fa68 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:41:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:41:52,066 [243] [DEBUG] [app] Starting request: urn:request:79d2bdd5-4af1-4929-8cbc-69afd64257cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:52,066 [230] [DEBUG] [app] Starting request: urn:request:e8eaec02-3f52-4dc7-8f2c-cef31e294683 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:41:52,066 [243] [DEBUG] [app] Ending request: urn:request:79d2bdd5-4af1-4929-8cbc-69afd64257cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:79d2bdd5-4af1-4929-8cbc-69afd64257cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:41:52,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:41:52,067 [230] [DEBUG] [app] Ending request: urn:request:e8eaec02-3f52-4dc7-8f2c-cef31e294683 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e8eaec02-3f52-4dc7-8f2c-cef31e294683', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:52,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:41:52,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:52,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:41:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:41:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:52,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:41:52,072 [235] [DEBUG] [app] Starting request: urn:request:281ba66f-3e1e-4962-bf32-cfb8f8124789 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:52,072 [231] [DEBUG] [app] Starting request: urn:request:7e7b9ee6-2c7a-4c4a-bdf5-b0d849f29593 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:41:52,072 [235] [DEBUG] [app] Ending request: urn:request:281ba66f-3e1e-4962-bf32-cfb8f8124789 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:281ba66f-3e1e-4962-bf32-cfb8f8124789', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:41:52,072 [231] [DEBUG] [app] Ending request: urn:request:7e7b9ee6-2c7a-4c4a-bdf5-b0d849f29593 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e7b9ee6-2c7a-4c4a-bdf5-b0d849f29593', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:41:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:52,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:41:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:41:52,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:41:52,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:41:52,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:52,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:41:52,090 [236] [DEBUG] [app] Ending request: urn:request:9cc6bb52-5d57-433c-85fb-a94f5fd8fa68 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9cc6bb52-5d57-433c-85fb-a94f5fd8fa68', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:52,090 [231] [DEBUG] [app] Ending request: urn:request:1cf1773b-3f6f-4a7c-b895-ff6fe6ba0361 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1cf1773b-3f6f-4a7c-b895-ff6fe6ba0361', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:41:52,090 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:52,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:41:52,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:41:52,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:41:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) builder stdout | 2023-01-17 05:41:53,134 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:41:53,146 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:41:53,146 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:41:53,156 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:41:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:41:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:28.963662+00:00 (in 33.999731 seconds) exportactionlogsworker stdout | 2023-01-17 05:41:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:54 UTC)" (scheduled at 2023-01-17 05:41:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:41:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 41, 54, 964382), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:41:54,983 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 54, 964382), True, datetime.datetime(2023, 1, 17, 5, 41, 54, 964382), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:41:54,987 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 41, 54, 964382), True, datetime.datetime(2023, 1, 17, 5, 41, 54, 964382), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 41, 54, 964382), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:41:54,990 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:41:54,991 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:46:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:41:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:01.636362+00:00 (in 5.989562 seconds) notificationworker stdout | 2023-01-17 05:41:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:05 UTC)" (scheduled at 2023-01-17 05:41:55.646354+00:00) notificationworker stdout | 2023-01-17 05:41:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:41:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 41, 55, 647238), True, datetime.datetime(2023, 1, 17, 5, 41, 55, 647238), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:41:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:41:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:41:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:41:59,439 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:41:59,443 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:42:00,356 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:42:00,863 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:42:00,886 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:42:01,149 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:42:01,205 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:42:01,207 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:42:01,209 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:42:01,212 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:42:01,215 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:42:01,424 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:42:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:05.646354+00:00 (in 4.009559 seconds) notificationworker stdout | 2023-01-17 05:42:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:01 UTC)" (scheduled at 2023-01-17 05:42:01.636362+00:00) notificationworker stdout | 2023-01-17 05:42:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:42:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:01 UTC)" executed successfully builder stdout | 2023-01-17 05:42:03,166 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:03,179 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:03,180 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:03,190 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:42:04,580 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:42:04,687 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:42:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:42:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:19.346120+00:00 (in 14.408789 seconds) repositorygcworker stdout | 2023-01-17 05:42:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:04 UTC)" (scheduled at 2023-01-17 05:42:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:42:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:42:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 4, 937769), True, datetime.datetime(2023, 1, 17, 5, 42, 4, 937769), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:42:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:42:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:42:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:42:05,192 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:42:05,195 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:42:05,197 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:42:05,562 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:42:05,565 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:42:05,567 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:42:05,570 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:42:05,574 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:42:05,577 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:42:05,580 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:42:05,582 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:42:05,584 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:42:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:15.646354+00:00 (in 9.999485 seconds) notificationworker stdout | 2023-01-17 05:42:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:15 UTC)" (scheduled at 2023-01-17 05:42:05.646354+00:00) notificationworker stdout | 2023-01-17 05:42:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 5, 647264), True, datetime.datetime(2023, 1, 17, 5, 42, 5, 647264), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:15 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:42:05,770 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:42:05,812 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:42:06,011 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:42:06,550 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:42:06,746 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} namespacegcworker stdout | 2023-01-17 05:42:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:42:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:41.148245+00:00 (in 34.206517 seconds) namespacegcworker stdout | 2023-01-17 05:42:06,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:47:06 UTC)" (scheduled at 2023-01-17 05:42:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:42:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 42, 6, 942137), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:42:06,954 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 42, 6, 942137), True, datetime.datetime(2023, 1, 17, 5, 42, 6, 942137), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:42:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 42, 6, 942137), True, datetime.datetime(2023, 1, 17, 5, 42, 6, 942137), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 42, 6, 942137), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:42:06,961 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:42:06,961 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:47:06 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:42:07,000 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:42:07,024 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:42:07,059 [231] [DEBUG] [app] Starting request: urn:request:b4ab7ba3-9d42-4c22-ad74-24242d0a1f56 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:07,060 [236] [DEBUG] [app] Starting request: urn:request:a0bdfd17-abfa-4513-aa23-3a6433e4863f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:07,065 [242] [DEBUG] [app] Starting request: urn:request:bf4741c8-f6da-4f0f-96ac-5f11e5335777 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:07,065 [230] [DEBUG] [app] Starting request: urn:request:f1049062-e9c8-4963-9e46-6bc59d0efacd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:07,065 [242] [DEBUG] [app] Ending request: urn:request:bf4741c8-f6da-4f0f-96ac-5f11e5335777 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf4741c8-f6da-4f0f-96ac-5f11e5335777', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:07,065 [230] [DEBUG] [app] Ending request: urn:request:f1049062-e9c8-4963-9e46-6bc59d0efacd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f1049062-e9c8-4963-9e46-6bc59d0efacd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:42:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:07,070 [235] [DEBUG] [app] Starting request: urn:request:58c905e7-e65e-445f-8a2e-0f3cb4286249 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:07,071 [233] [DEBUG] [app] Starting request: urn:request:44976cc5-d297-49c6-94d4-048ab801944f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:07,071 [235] [DEBUG] [app] Ending request: urn:request:58c905e7-e65e-445f-8a2e-0f3cb4286249 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58c905e7-e65e-445f-8a2e-0f3cb4286249', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:07,071 [233] [DEBUG] [app] Ending request: urn:request:44976cc5-d297-49c6-94d4-048ab801944f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:44976cc5-d297-49c6-94d4-048ab801944f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:42:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:07,087 [231] [DEBUG] [app] Ending request: urn:request:b4ab7ba3-9d42-4c22-ad74-24242d0a1f56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b4ab7ba3-9d42-4c22-ad74-24242d0a1f56', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:42:07,088 [236] [DEBUG] [app] Ending request: urn:request:a0bdfd17-abfa-4513-aa23-3a6433e4863f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0bdfd17-abfa-4513-aa23-3a6433e4863f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:07,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:42:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" globalpromstats stdout | 2023-01-17 05:42:07,574 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:42:08,175 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:42:08,188 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:42:08,528 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:42:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:42:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:40.738568+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2023-01-17 05:42:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:40 UTC)" (scheduled at 2023-01-17 05:42:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:42:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 42, 10, 739462), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:42:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:42:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:42:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:40 UTC)" executed successfully builder stdout | 2023-01-17 05:42:13,192 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:13,204 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:13,204 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:13,215 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:25.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:42:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:25 UTC)" (scheduled at 2023-01-17 05:42:15.646354+00:00) notificationworker stdout | 2023-01-17 05:42:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 15, 647223), True, datetime.datetime(2023, 1, 17, 5, 42, 15, 647223), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:42:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:19.537472+00:00 (in 2.600010 seconds) securityworker stdout | 2023-01-17 05:42:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:46 UTC)" (scheduled at 2023-01-17 05:42:16.936988+00:00) securityworker stdout | 2023-01-17 05:42:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:42:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:42:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:42:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:04.936835+00:00 (in 45.590253 seconds) repositorygcworker stdout | 2023-01-17 05:42:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:19 UTC)" (scheduled at 2023-01-17 05:42:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:42:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:42:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:42:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:46.936988+00:00 (in 27.399056 seconds) securityworker stdout | 2023-01-17 05:42:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:49 UTC)" (scheduled at 2023-01-17 05:42:19.537472+00:00) securityworker stdout | 2023-01-17 05:42:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:42:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:42:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:19,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:42:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:42:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:42:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:42:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:42:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:42:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:42:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:51.348605+00:00 (in 29.999497 seconds) gcworker stdout | 2023-01-17 05:42:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:51 UTC)" (scheduled at 2023-01-17 05:42:21.348605+00:00) gcworker stdout | 2023-01-17 05:42:21,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:42:21,360 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724541360, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:42:21,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:42:21,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:42:21,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:42:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:42:22,060 [231] [DEBUG] [app] Starting request: urn:request:e53adeea-b6ac-4a03-bb77-25b2967c7ff9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:22,060 [235] [DEBUG] [app] Starting request: urn:request:fd45f7ca-c640-4d86-a6f7-8c4f8bcf3bac (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:22,065 [243] [DEBUG] [app] Starting request: urn:request:6d52ff46-4fc6-43b9-abab-78b1077a934b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:22,065 [230] [DEBUG] [app] Starting request: urn:request:11d0e8e6-8cf0-4c83-85be-d252450daabb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:22,065 [243] [DEBUG] [app] Ending request: urn:request:6d52ff46-4fc6-43b9-abab-78b1077a934b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d52ff46-4fc6-43b9-abab-78b1077a934b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:42:22,066 [230] [DEBUG] [app] Ending request: urn:request:11d0e8e6-8cf0-4c83-85be-d252450daabb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11d0e8e6-8cf0-4c83-85be-d252450daabb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:42:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:42:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:22,071 [233] [DEBUG] [app] Starting request: urn:request:2fce5379-1f6a-4f1d-8811-77b468e05112 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:22,071 [236] [DEBUG] [app] Starting request: urn:request:76aebdf3-0875-4fb4-82b4-be78b0c782e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:22,071 [233] [DEBUG] [app] Ending request: urn:request:2fce5379-1f6a-4f1d-8811-77b468e05112 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2fce5379-1f6a-4f1d-8811-77b468e05112', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:22,072 [236] [DEBUG] [app] Ending request: urn:request:76aebdf3-0875-4fb4-82b4-be78b0c782e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76aebdf3-0875-4fb4-82b4-be78b0c782e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:42:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:22,088 [231] [DEBUG] [app] Ending request: urn:request:e53adeea-b6ac-4a03-bb77-25b2967c7ff9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e53adeea-b6ac-4a03-bb77-25b2967c7ff9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:22,088 [235] [DEBUG] [app] Ending request: urn:request:fd45f7ca-c640-4d86-a6f7-8c4f8bcf3bac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd45f7ca-c640-4d86-a6f7-8c4f8bcf3bac', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:22,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:42:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:42:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:42:23,217 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:23,229 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:23,229 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:23,240 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:35.646354+00:00 (in 9.999536 seconds) notificationworker stdout | 2023-01-17 05:42:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:35 UTC)" (scheduled at 2023-01-17 05:42:25.646354+00:00) notificationworker stdout | 2023-01-17 05:42:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 25, 647250), True, datetime.datetime(2023, 1, 17, 5, 42, 25, 647250), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:42:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:42:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:35.853396+00:00 (in 6.889196 seconds) exportactionlogsworker stdout | 2023-01-17 05:42:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:28 UTC)" (scheduled at 2023-01-17 05:42:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:42:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:42:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:28 UTC)" executed successfully gcworker stdout | 2023-01-17 05:42:29,473 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:42:29,475 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:42:30,392 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:42:30,899 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:42:30,911 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:42:31,155 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:42:31,211 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:42:31,213 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:42:31,216 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:42:31,218 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:42:31,222 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:42:31,459 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:42:33,250 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:33,263 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:33,263 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:33,273 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:42:34,615 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:42:34,710 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:42:35,199 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:42:35,201 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:42:35,204 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:42:35,573 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:42:35,576 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:42:35,578 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:42:35,582 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:42:35,584 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:42:35,586 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:42:35,589 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:42:35,591 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:42:35,594 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:42:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:45.646354+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 05:42:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:45 UTC)" (scheduled at 2023-01-17 05:42:35.646354+00:00) notificationworker stdout | 2023-01-17 05:42:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 35, 647242), True, datetime.datetime(2023, 1, 17, 5, 42, 35, 647242), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:45 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:42:35,805 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:42:35,839 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} exportactionlogsworker stdout | 2023-01-17 05:42:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:42:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:28.963662+00:00 (in 53.109598 seconds) exportactionlogsworker stdout | 2023-01-17 05:42:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:35 UTC)" (scheduled at 2023-01-17 05:42:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:42:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:42:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 35, 854498), True, datetime.datetime(2023, 1, 17, 5, 42, 35, 854498), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:42:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:42:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:42:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:35 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:42:36,046 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:42:36,557 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:42:36,782 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:42:37,035 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:42:37,042 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:42:37,059 [235] [DEBUG] [app] Starting request: urn:request:cab90052-c8d6-4b33-9e2f-8a9078ecd808 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:37,059 [236] [DEBUG] [app] Starting request: urn:request:592bbd30-4c09-46ac-90fd-86c62edacad3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:37,064 [230] [DEBUG] [app] Starting request: urn:request:6927975f-d94b-4252-81b0-d4ee0db9c5ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:37,065 [242] [DEBUG] [app] Starting request: urn:request:3552b0d6-a3b6-42c4-af06-df588e48a77d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:37,065 [230] [DEBUG] [app] Ending request: urn:request:6927975f-d94b-4252-81b0-d4ee0db9c5ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6927975f-d94b-4252-81b0-d4ee0db9c5ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:42:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:42:37,065 [242] [DEBUG] [app] Ending request: urn:request:3552b0d6-a3b6-42c4-af06-df588e48a77d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3552b0d6-a3b6-42c4-af06-df588e48a77d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:42:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:37,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:37,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:37,069 [231] [DEBUG] [app] Starting request: urn:request:85e191ae-bb89-4dd9-a400-a520eb423cda (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:37,070 [233] [DEBUG] [app] Starting request: urn:request:e17f73d5-7635-4d96-8b05-78344efa181a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:37,070 [231] [DEBUG] [app] Ending request: urn:request:85e191ae-bb89-4dd9-a400-a520eb423cda (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:85e191ae-bb89-4dd9-a400-a520eb423cda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:42:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:37,070 [233] [DEBUG] [app] Ending request: urn:request:e17f73d5-7635-4d96-8b05-78344efa181a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e17f73d5-7635-4d96-8b05-78344efa181a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:37,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:37,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:37,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:37,087 [236] [DEBUG] [app] Ending request: urn:request:592bbd30-4c09-46ac-90fd-86c62edacad3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:592bbd30-4c09-46ac-90fd-86c62edacad3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:37,087 [235] [DEBUG] [app] Ending request: urn:request:cab90052-c8d6-4b33-9e2f-8a9078ecd808 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cab90052-c8d6-4b33-9e2f-8a9078ecd808', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:42:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:42:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:42:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:37.537407+00:00 (in 0.280343 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:42:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:37 UTC)" (scheduled at 2023-01-17 05:42:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:42:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:42:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 37, 257442), True, datetime.datetime(2023, 1, 17, 5, 42, 37, 257442), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:42:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:42:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:42:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:42:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:42:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:37.256652+00:00 (in 59.718864 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:42:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:37 UTC)" (scheduled at 2023-01-17 05:42:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:42:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:42:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:42:37,608 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:42:38,181 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:42:38,223 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:42:38,564 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:42:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:42:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:10.738568+00:00 (in 29.999521 seconds) buildlogsarchiver stdout | 2023-01-17 05:42:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:10 UTC)" (scheduled at 2023-01-17 05:42:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:42:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 42, 40, 739460), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:42:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:42:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:42:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:42:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:42:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:42.941536+00:00 (in 1.792773 seconds) namespacegcworker stdout | 2023-01-17 05:42:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:41 UTC)" (scheduled at 2023-01-17 05:42:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:42:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:42:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 41, 149207), True, datetime.datetime(2023, 1, 17, 5, 42, 41, 149207), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:42:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:42:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:42:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:42:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:42:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:41.148245+00:00 (in 58.206208 seconds) namespacegcworker stdout | 2023-01-17 05:42:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:42 UTC)" (scheduled at 2023-01-17 05:42:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:42:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:42:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:43:42 UTC)" executed successfully builder stdout | 2023-01-17 05:42:43,283 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:43,296 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:43,296 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:43,307 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:55.646354+00:00 (in 9.999524 seconds) notificationworker stdout | 2023-01-17 05:42:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:55 UTC)" (scheduled at 2023-01-17 05:42:45.646354+00:00) notificationworker stdout | 2023-01-17 05:42:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 45, 647222), True, datetime.datetime(2023, 1, 17, 5, 42, 45, 647222), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:42:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:42:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:42:49.537472+00:00 (in 2.600039 seconds) securityworker stdout | 2023-01-17 05:42:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:16 UTC)" (scheduled at 2023-01-17 05:42:46.936988+00:00) securityworker stdout | 2023-01-17 05:42:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:42:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:42:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:42:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:16.936988+00:00 (in 27.399075 seconds) securityworker stdout | 2023-01-17 05:42:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:19 UTC)" (scheduled at 2023-01-17 05:42:49.537472+00:00) securityworker stdout | 2023-01-17 05:42:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:42:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:42:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:42:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:42:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:42:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:42:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:42:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:42:49,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:42:49,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:42:49,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:42:49,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:42:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:42:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:42:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:42:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:21.348605+00:00 (in 29.999553 seconds) gcworker stdout | 2023-01-17 05:42:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:21 UTC)" (scheduled at 2023-01-17 05:42:51.348605+00:00) gcworker stdout | 2023-01-17 05:42:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724571349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:42:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:42:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:42:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:42:52,060 [231] [DEBUG] [app] Starting request: urn:request:f51074da-6215-48d1-a974-1f52ddfe31c9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:52,061 [235] [DEBUG] [app] Starting request: urn:request:bf75a6bd-559b-4b5a-bf76-c3a16f4f3d01 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:42:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:42:52,065 [230] [DEBUG] [app] Starting request: urn:request:01bc794f-01b6-4534-b311-e5d3edde38ee (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:52,066 [230] [DEBUG] [app] Ending request: urn:request:01bc794f-01b6-4534-b311-e5d3edde38ee (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01bc794f-01b6-4534-b311-e5d3edde38ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:42:52,066 [243] [DEBUG] [app] Starting request: urn:request:7cc40a45-c9b3-4675-99e9-044128d22c02 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:42:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:42:52,066 [243] [DEBUG] [app] Ending request: urn:request:7cc40a45-c9b3-4675-99e9-044128d22c02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7cc40a45-c9b3-4675-99e9-044128d22c02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:42:52,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:42:52,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:42:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:42:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:42:52,071 [233] [DEBUG] [app] Starting request: urn:request:852e7513-bee3-4f8b-96f6-db86c2744e4b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:52,071 [233] [DEBUG] [app] Ending request: urn:request:852e7513-bee3-4f8b-96f6-db86c2744e4b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:852e7513-bee3-4f8b-96f6-db86c2744e4b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:52,071 [236] [DEBUG] [app] Starting request: urn:request:b0916341-a85e-410b-820d-7cc6770101bb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:42:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:42:52,072 [236] [DEBUG] [app] Ending request: urn:request:b0916341-a85e-410b-820d-7cc6770101bb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b0916341-a85e-410b-820d-7cc6770101bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:42:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:42:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:42:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:42:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:42:52,088 [231] [DEBUG] [app] Ending request: urn:request:f51074da-6215-48d1-a974-1f52ddfe31c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f51074da-6215-48d1-a974-1f52ddfe31c9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:42:52,088 [235] [DEBUG] [app] Ending request: urn:request:bf75a6bd-559b-4b5a-bf76-c3a16f4f3d01 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf75a6bd-559b-4b5a-bf76-c3a16f4f3d01', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:42:52,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:42:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:42:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:42:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:42:53,308 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:42:53,320 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:42:53,320 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:42:53,331 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:42:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:01.636362+00:00 (in 5.989542 seconds) notificationworker stdout | 2023-01-17 05:42:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:05 UTC)" (scheduled at 2023-01-17 05:42:55.646354+00:00) notificationworker stdout | 2023-01-17 05:42:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:42:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 42, 55, 647251), True, datetime.datetime(2023, 1, 17, 5, 42, 55, 647251), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:42:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:42:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:42:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:42:59,485 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:42:59,507 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:43:00,398 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:43:00,935 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:43:00,946 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:43:01,174 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:43:01,217 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:43:01,220 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:43:01,223 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:43:01,225 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:43:01,228 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:43:01,473 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:43:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:05.646354+00:00 (in 4.009511 seconds) notificationworker stdout | 2023-01-17 05:43:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:01 UTC)" (scheduled at 2023-01-17 05:43:01.636362+00:00) notificationworker stdout | 2023-01-17 05:43:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:43:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:01 UTC)" executed successfully builder stdout | 2023-01-17 05:43:03,341 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:03,354 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:03,354 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:03,364 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:43:04,651 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:43:04,744 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:43:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:43:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:19.346120+00:00 (in 14.408829 seconds) repositorygcworker stdout | 2023-01-17 05:43:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:04 UTC)" (scheduled at 2023-01-17 05:43:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:43:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:43:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 4, 937760), True, datetime.datetime(2023, 1, 17, 5, 43, 4, 937760), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:43:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:43:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:43:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:43:05,206 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:43:05,208 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:43:05,211 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:43:05,583 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:43:05,585 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:43:05,587 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:43:05,590 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:43:05,592 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:43:05,595 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:43:05,598 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:43:05,600 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:43:05,602 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:43:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:15.646354+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 05:43:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:15 UTC)" (scheduled at 2023-01-17 05:43:05.646354+00:00) notificationworker stdout | 2023-01-17 05:43:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 5, 647193), True, datetime.datetime(2023, 1, 17, 5, 43, 5, 647193), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:15 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:43:05,841 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:43:05,873 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:43:06,081 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:43:06,577 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:43:06,812 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:43:07,060 [236] [DEBUG] [app] Starting request: urn:request:9faefa26-746f-4edf-a929-6eea554b2d89 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:43:07,060 [235] [DEBUG] [app] Starting request: urn:request:05ab5340-3258-4ae7-b7c0-610a87399800 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} securityscanningnotificationworker stdout | 2023-01-17 05:43:07,061 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-web stdout | 2023-01-17 05:43:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:07,065 [241] [DEBUG] [app] Starting request: urn:request:317bf1b1-b12c-4b60-8303-d90446bf6cfc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:07,065 [242] [DEBUG] [app] Starting request: urn:request:cb45c4ab-54cf-4ba5-be4c-704c9d53e972 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:07,066 [241] [DEBUG] [app] Ending request: urn:request:317bf1b1-b12c-4b60-8303-d90446bf6cfc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:317bf1b1-b12c-4b60-8303-d90446bf6cfc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:07,066 [242] [DEBUG] [app] Ending request: urn:request:cb45c4ab-54cf-4ba5-be4c-704c9d53e972 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb45c4ab-54cf-4ba5-be4c-704c9d53e972', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:43:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:43:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:07,071 [233] [DEBUG] [app] Starting request: urn:request:3659282b-3540-4a13-8bf9-3ead1a7b7095 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:07,071 [231] [DEBUG] [app] Starting request: urn:request:648ad82b-fe0a-4a97-ae7b-879580c59aea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:07,071 [233] [DEBUG] [app] Ending request: urn:request:3659282b-3540-4a13-8bf9-3ead1a7b7095 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3659282b-3540-4a13-8bf9-3ead1a7b7095', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:07,072 [231] [DEBUG] [app] Ending request: urn:request:648ad82b-fe0a-4a97-ae7b-879580c59aea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:648ad82b-fe0a-4a97-ae7b-879580c59aea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql repositoryactioncounter stdout | 2023-01-17 05:43:07,076 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} gunicorn-web stdout | 2023-01-17 05:43:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:07,088 [235] [DEBUG] [app] Ending request: urn:request:05ab5340-3258-4ae7-b7c0-610a87399800 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05ab5340-3258-4ae7-b7c0-610a87399800', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:07,088 [236] [DEBUG] [app] Ending request: urn:request:9faefa26-746f-4edf-a929-6eea554b2d89 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9faefa26-746f-4edf-a929-6eea554b2d89', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:43:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" globalpromstats stdout | 2023-01-17 05:43:07,627 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:43:08,203 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:43:08,240 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:43:08,596 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:43:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:43:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:40.738568+00:00 (in 29.999546 seconds) buildlogsarchiver stdout | 2023-01-17 05:43:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:40 UTC)" (scheduled at 2023-01-17 05:43:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:43:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 43, 10, 739428), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:43:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:43:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:43:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:40 UTC)" executed successfully builder stdout | 2023-01-17 05:43:13,372 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:13,385 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:13,385 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:13,395 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:25.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:43:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:25 UTC)" (scheduled at 2023-01-17 05:43:15.646354+00:00) notificationworker stdout | 2023-01-17 05:43:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 15, 647249), True, datetime.datetime(2023, 1, 17, 5, 43, 15, 647249), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:19.537472+00:00 (in 2.600047 seconds) securityworker stdout | 2023-01-17 05:43:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:46 UTC)" (scheduled at 2023-01-17 05:43:16.936988+00:00) securityworker stdout | 2023-01-17 05:43:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:16,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:43:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:43:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:04.936835+00:00 (in 45.590230 seconds) repositorygcworker stdout | 2023-01-17 05:43:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:19 UTC)" (scheduled at 2023-01-17 05:43:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:43:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:43:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:46.936988+00:00 (in 27.399088 seconds) securityworker stdout | 2023-01-17 05:43:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:49 UTC)" (scheduled at 2023-01-17 05:43:19.537472+00:00) securityworker stdout | 2023-01-17 05:43:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:43:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:43:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:43:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:43:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:43:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:43:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:43:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:43:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:51.348605+00:00 (in 29.999541 seconds) gcworker stdout | 2023-01-17 05:43:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:51 UTC)" (scheduled at 2023-01-17 05:43:21.348605+00:00) gcworker stdout | 2023-01-17 05:43:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724601349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:43:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:43:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:43:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:43:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:43:22,060 [231] [DEBUG] [app] Starting request: urn:request:e3063ad1-5120-4a84-9c02-31a96d873372 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:43:22,060 [236] [DEBUG] [app] Starting request: urn:request:8a5e2e68-39cc-469c-99eb-19d774e69af1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:43:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:22,065 [230] [DEBUG] [app] Starting request: urn:request:7a7d7196-6c83-41ab-bf33-7c3b09eba81e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:22,065 [242] [DEBUG] [app] Starting request: urn:request:640cf62e-c609-445b-bc4d-119b50ed5f93 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:22,065 [230] [DEBUG] [app] Ending request: urn:request:7a7d7196-6c83-41ab-bf33-7c3b09eba81e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7a7d7196-6c83-41ab-bf33-7c3b09eba81e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:43:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:43:22,066 [242] [DEBUG] [app] Ending request: urn:request:640cf62e-c609-445b-bc4d-119b50ed5f93 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:640cf62e-c609-445b-bc4d-119b50ed5f93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:43:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:43:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:22,070 [235] [DEBUG] [app] Starting request: urn:request:82e80772-36d0-4b60-a1bf-d18638b39edc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:22,070 [233] [DEBUG] [app] Starting request: urn:request:b3114201-07bf-441c-8b1e-e3fd9d852cb0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:22,070 [235] [DEBUG] [app] Ending request: urn:request:82e80772-36d0-4b60-a1bf-d18638b39edc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82e80772-36d0-4b60-a1bf-d18638b39edc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:22,071 [233] [DEBUG] [app] Ending request: urn:request:b3114201-07bf-441c-8b1e-e3fd9d852cb0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3114201-07bf-441c-8b1e-e3fd9d852cb0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:43:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:22,088 [236] [DEBUG] [app] Ending request: urn:request:8a5e2e68-39cc-469c-99eb-19d774e69af1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a5e2e68-39cc-469c-99eb-19d774e69af1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:22,088 [231] [DEBUG] [app] Ending request: urn:request:e3063ad1-5120-4a84-9c02-31a96d873372 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3063ad1-5120-4a84-9c02-31a96d873372', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:43:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:43:23,405 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:23,418 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:23,418 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:23,429 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:35.646354+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 05:43:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:35 UTC)" (scheduled at 2023-01-17 05:43:25.646354+00:00) notificationworker stdout | 2023-01-17 05:43:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 25, 647208), True, datetime.datetime(2023, 1, 17, 5, 43, 25, 647208), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:43:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:43:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:35.853396+00:00 (in 6.889192 seconds) exportactionlogsworker stdout | 2023-01-17 05:43:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:28 UTC)" (scheduled at 2023-01-17 05:43:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:43:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:43:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:29,493 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:43:29,542 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:43:30,403 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:43:30,970 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:43:30,981 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:43:31,187 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:43:31,222 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:43:31,227 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:43:31,229 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:43:31,232 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:43:31,234 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:43:31,508 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:43:33,430 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:33,442 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:33,443 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:33,453 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:43:34,658 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:43:34,752 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:43:35,214 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:43:35,217 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:43:35,219 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:43:35,590 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:43:35,592 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:43:35,595 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:43:35,597 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:43:35,600 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:43:35,603 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:43:35,605 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:43:35,607 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:43:35,609 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} notificationworker stdout | 2023-01-17 05:43:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:36.635997+00:00 (in 0.989226 seconds) notificationworker stdout | 2023-01-17 05:43:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:45 UTC)" (scheduled at 2023-01-17 05:43:35.646354+00:00) notificationworker stdout | 2023-01-17 05:43:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 35, 647156), True, datetime.datetime(2023, 1, 17, 5, 43, 35, 647156), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:43:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:43:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:28.963662+00:00 (in 53.109776 seconds) exportactionlogsworker stdout | 2023-01-17 05:43:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:35 UTC)" (scheduled at 2023-01-17 05:43:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:43:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:43:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 35, 854278), True, datetime.datetime(2023, 1, 17, 5, 43, 35, 854278), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:43:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:43:35,864 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:43:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:43:35,877 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:43:35,901 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:43:36,112 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:43:36,613 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} notificationworker stdout | 2023-01-17 05:43:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:45.646354+00:00 (in 9.009918 seconds) notificationworker stdout | 2023-01-17 05:43:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:48:36 UTC)" (scheduled at 2023-01-17 05:43:36.635997+00:00) notificationworker stdout | 2023-01-17 05:43:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 43, 36, 636783), 'notification/%']) notificationworker stdout | 2023-01-17 05:43:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 43, 36, 636783), True, datetime.datetime(2023, 1, 17, 5, 43, 36, 636783), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:43:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 43, 36, 636783), True, datetime.datetime(2023, 1, 17, 5, 43, 36, 636783), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 43, 36, 636783), 'notification/%']) notificationworker stdout | 2023-01-17 05:43:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:48:36 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:43:36,847 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:43:37,060 [235] [DEBUG] [app] Starting request: urn:request:e9437fec-226d-4533-ab8e-f10a6d94742b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:43:37,060 [231] [DEBUG] [app] Starting request: urn:request:9cf005c2-f71e-40d5-b46e-7552419ae4ae (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:43:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:37,065 [230] [DEBUG] [app] Starting request: urn:request:656371a3-30e4-468b-a12f-fe92579f9698 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:37,066 [243] [DEBUG] [app] Starting request: urn:request:27d22b0d-dfc1-4f0e-ab18-5a54c8d2c3fc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:37,066 [230] [DEBUG] [app] Ending request: urn:request:656371a3-30e4-468b-a12f-fe92579f9698 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:656371a3-30e4-468b-a12f-fe92579f9698', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:43:37,066 [243] [DEBUG] [app] Ending request: urn:request:27d22b0d-dfc1-4f0e-ab18-5a54c8d2c3fc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27d22b0d-dfc1-4f0e-ab18-5a54c8d2c3fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:43:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:43:37,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:43:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:37,071 [236] [DEBUG] [app] Starting request: urn:request:4c9f1b5b-853e-4d06-a615-68dad5a5e1f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:37,071 [235] [DEBUG] [app] Starting request: urn:request:f366a665-930b-426a-b4e3-f2a04bea8d04 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:37,072 [236] [DEBUG] [app] Ending request: urn:request:4c9f1b5b-853e-4d06-a615-68dad5a5e1f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c9f1b5b-853e-4d06-a615-68dad5a5e1f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:37,072 [235] [DEBUG] [app] Ending request: urn:request:f366a665-930b-426a-b4e3-f2a04bea8d04 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f366a665-930b-426a-b4e3-f2a04bea8d04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:37,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:37,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:37,088 [235] [DEBUG] [app] Ending request: urn:request:e9437fec-226d-4533-ab8e-f10a6d94742b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e9437fec-226d-4533-ab8e-f10a6d94742b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:37,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:43:37,089 [231] [DEBUG] [app] Ending request: urn:request:9cf005c2-f71e-40d5-b46e-7552419ae4ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9cf005c2-f71e-40d5-b46e-7552419ae4ae', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:43:37,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:43:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" repositoryactioncounter stdout | 2023-01-17 05:43:37,092 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:43:37,095 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:43:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:43:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:37.537407+00:00 (in 0.280370 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:43:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:37 UTC)" (scheduled at 2023-01-17 05:43:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:43:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:43:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 37, 257406), True, datetime.datetime(2023, 1, 17, 5, 43, 37, 257406), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:43:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:43:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:43:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:43:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:43:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:37.256652+00:00 (in 59.718890 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:43:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:37 UTC)" (scheduled at 2023-01-17 05:43:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:43:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:43:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:43:37,663 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:43:38,238 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:43:38,275 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:43:38,632 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} globalpromstats stdout | 2023-01-17 05:43:39,344 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run globalpromstats stdout | 2023-01-17 05:43:39,344 [67] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2023-01-17 05:43:39 UTC)" (scheduled at 2023-01-17 05:43:39.344159+00:00) globalpromstats stdout | 2023-01-17 05:43:39,344 [67] [DEBUG] [__main__] Attempting to report stats globalpromstats stdout | 2023-01-17 05:43:39,344 [67] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:43:39,345 [67] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ... globalpromstats stdout | 2023-01-17 05:43:39,345 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:43:39.344159+00:00 (in 3599.999686 seconds) globalpromstats stdout | 2023-01-17 05:43:39,347 [67] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2023-01-17 05:43:39,347 [67] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:43:39,347 [67] [DEBUG] [__main__] Reporting global stats globalpromstats stdout | 2023-01-17 05:43:39,347 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2023-01-17 05:43:39,358 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False]) globalpromstats stdout | 2023-01-17 05:43:39,361 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True]) globalpromstats stdout | 2023-01-17 05:43:39,365 [67] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', []) globalpromstats stdout | 2023-01-17 05:43:39,368 [67] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 05:43:39,368 [67] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:43:39,368 [67] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'. globalpromstats stdout | 2023-01-17 05:43:39,369 [67] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS globalpromstats stdout | 2023-01-17 05:43:39,369 [67] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2023-01-17 06:43:39 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:43:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:43:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:10.738568+00:00 (in 29.999569 seconds) buildlogsarchiver stdout | 2023-01-17 05:43:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:10 UTC)" (scheduled at 2023-01-17 05:43:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:43:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 43, 40, 739417), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:43:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:43:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:43:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:43:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:43:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:42.941536+00:00 (in 1.792811 seconds) namespacegcworker stdout | 2023-01-17 05:43:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:41 UTC)" (scheduled at 2023-01-17 05:43:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:43:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:43:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 41, 149122), True, datetime.datetime(2023, 1, 17, 5, 43, 41, 149122), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:43:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:43:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:43:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:43:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:43:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:41.148245+00:00 (in 58.206231 seconds) namespacegcworker stdout | 2023-01-17 05:43:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:42 UTC)" (scheduled at 2023-01-17 05:43:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:43:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:43:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:44:42 UTC)" executed successfully builder stdout | 2023-01-17 05:43:43,463 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:43,476 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:43,476 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:43,487 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:55.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:43:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:55 UTC)" (scheduled at 2023-01-17 05:43:45.646354+00:00) notificationworker stdout | 2023-01-17 05:43:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 45, 647261), True, datetime.datetime(2023, 1, 17, 5, 43, 45, 647261), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:43:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:43:49.537472+00:00 (in 2.600051 seconds) securityworker stdout | 2023-01-17 05:43:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:16 UTC)" (scheduled at 2023-01-17 05:43:46.936988+00:00) securityworker stdout | 2023-01-17 05:43:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:43:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:43:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:16.936988+00:00 (in 27.399065 seconds) securityworker stdout | 2023-01-17 05:43:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:19 UTC)" (scheduled at 2023-01-17 05:43:49.537472+00:00) securityworker stdout | 2023-01-17 05:43:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:43:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:43:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:43:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:43:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:43:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:43:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:43:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:43:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:43:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:43:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:43:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:43:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:43:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:43:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:43:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:21.348605+00:00 (in 29.999567 seconds) gcworker stdout | 2023-01-17 05:43:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:21 UTC)" (scheduled at 2023-01-17 05:43:51.348605+00:00) gcworker stdout | 2023-01-17 05:43:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724631349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:43:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:43:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:43:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:43:52,060 [231] [DEBUG] [app] Starting request: urn:request:18ed98bf-2c78-49fd-baac-87d0b8cf37ac (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:43:52,060 [235] [DEBUG] [app] Starting request: urn:request:ec9682bc-d4c2-4cb2-a1c4-daf6df399190 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:43:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:43:52,065 [230] [DEBUG] [app] Starting request: urn:request:333e0da0-363a-4491-a7cb-b0d6208e47b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,065 [243] [DEBUG] [app] Starting request: urn:request:912c1531-20ae-4dd1-a76a-9655a8d24af9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,066 [230] [DEBUG] [app] Ending request: urn:request:333e0da0-363a-4491-a7cb-b0d6208e47b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:333e0da0-363a-4491-a7cb-b0d6208e47b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,066 [243] [DEBUG] [app] Ending request: urn:request:912c1531-20ae-4dd1-a76a-9655a8d24af9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:912c1531-20ae-4dd1-a76a-9655a8d24af9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:43:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:43:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:43:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:43:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:43:52,071 [236] [DEBUG] [app] Starting request: urn:request:cab815c9-a9a9-4146-9761-64e6e52b31da (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:52,071 [231] [DEBUG] [app] Starting request: urn:request:8ba9094a-bf2a-4b2a-b0a1-ba33b08f0b1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:43:52,071 [236] [DEBUG] [app] Ending request: urn:request:cab815c9-a9a9-4146-9761-64e6e52b31da (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cab815c9-a9a9-4146-9761-64e6e52b31da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:43:52,071 [231] [DEBUG] [app] Ending request: urn:request:8ba9094a-bf2a-4b2a-b0a1-ba33b08f0b1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ba9094a-bf2a-4b2a-b0a1-ba33b08f0b1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:52,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:43:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:43:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:43:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:43:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:43:52,088 [231] [DEBUG] [app] Ending request: urn:request:18ed98bf-2c78-49fd-baac-87d0b8cf37ac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:18ed98bf-2c78-49fd-baac-87d0b8cf37ac', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:52,088 [235] [DEBUG] [app] Ending request: urn:request:ec9682bc-d4c2-4cb2-a1c4-daf6df399190 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec9682bc-d4c2-4cb2-a1c4-daf6df399190', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:43:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:43:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:43:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:43:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:43:53,493 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:43:53,506 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:43:53,506 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:43:53,517 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:43:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:01.636362+00:00 (in 5.989519 seconds) notificationworker stdout | 2023-01-17 05:43:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:05 UTC)" (scheduled at 2023-01-17 05:43:55.646354+00:00) notificationworker stdout | 2023-01-17 05:43:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:43:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 43, 55, 647282), True, datetime.datetime(2023, 1, 17, 5, 43, 55, 647282), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:43:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:43:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:43:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:43:59,500 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:43:59,573 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:44:00,439 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:44:00,994 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:44:01,016 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:44:01,222 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} gunicorn-web stdout | 2023-01-17 05:44:01,229 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:44:01,234 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:44:01,236 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:44:01,239 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:44:01,241 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} securityworker stdout | 2023-01-17 05:44:01,534 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:44:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:05.646354+00:00 (in 4.009484 seconds) notificationworker stdout | 2023-01-17 05:44:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:01 UTC)" (scheduled at 2023-01-17 05:44:01.636362+00:00) notificationworker stdout | 2023-01-17 05:44:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:44:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:01 UTC)" executed successfully builder stdout | 2023-01-17 05:44:03,527 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:03,539 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:03,539 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:03,550 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:44:04,678 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:44:04,758 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:44:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:44:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:19.346120+00:00 (in 14.408829 seconds) repositorygcworker stdout | 2023-01-17 05:44:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:04 UTC)" (scheduled at 2023-01-17 05:44:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:44:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:44:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 4, 937754), True, datetime.datetime(2023, 1, 17, 5, 44, 4, 937754), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:44:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:44:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:44:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:44:05,222 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:44:05,224 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:44:05,227 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:44:05,597 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:44:05,599 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:44:05,603 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:44:05,605 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:44:05,607 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:44:05,611 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:44:05,613 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:44:05,615 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:44:05,618 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} notificationworker stdout | 2023-01-17 05:44:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:15.646354+00:00 (in 9.999544 seconds) notificationworker stdout | 2023-01-17 05:44:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:15 UTC)" (scheduled at 2023-01-17 05:44:05.646354+00:00) notificationworker stdout | 2023-01-17 05:44:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 5, 647223), True, datetime.datetime(2023, 1, 17, 5, 44, 5, 647223), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:15 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:44:05,907 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:44:05,909 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:44:06,139 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:44:06,649 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:44:06,857 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:44:07,060 [233] [DEBUG] [app] Starting request: urn:request:3aeec716-1fee-4ace-b4ba-f645a3b8e5c5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:07,060 [235] [DEBUG] [app] Starting request: urn:request:605b9321-d9ef-425b-9208-274c3a0e0d98 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:07,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:07,065 [239] [DEBUG] [app] Starting request: urn:request:cc82418c-02ac-45c6-82da-e5aa40a8f530 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:07,065 [242] [DEBUG] [app] Starting request: urn:request:9f4f80ed-8b0e-4b6f-8366-71a9c1f6adeb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:07,066 [239] [DEBUG] [app] Ending request: urn:request:cc82418c-02ac-45c6-82da-e5aa40a8f530 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc82418c-02ac-45c6-82da-e5aa40a8f530', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:44:07,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:07,066 [242] [DEBUG] [app] Ending request: urn:request:9f4f80ed-8b0e-4b6f-8366-71a9c1f6adeb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9f4f80ed-8b0e-4b6f-8366-71a9c1f6adeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:07,070 [231] [DEBUG] [app] Starting request: urn:request:1122294d-e750-47fa-940e-465e773b1079 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:07,071 [236] [DEBUG] [app] Starting request: urn:request:73ac686d-62d2-443d-9cef-d634206a9b78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:07,071 [231] [DEBUG] [app] Ending request: urn:request:1122294d-e750-47fa-940e-465e773b1079 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1122294d-e750-47fa-940e-465e773b1079', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:07,071 [236] [DEBUG] [app] Ending request: urn:request:73ac686d-62d2-443d-9cef-d634206a9b78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73ac686d-62d2-443d-9cef-d634206a9b78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:07,088 [235] [DEBUG] [app] Ending request: urn:request:605b9321-d9ef-425b-9208-274c3a0e0d98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:605b9321-d9ef-425b-9208-274c3a0e0d98', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:07,088 [233] [DEBUG] [app] Ending request: urn:request:3aeec716-1fee-4ace-b4ba-f645a3b8e5c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3aeec716-1fee-4ace-b4ba-f645a3b8e5c5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:07,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:44:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:44:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) repositoryactioncounter stdout | 2023-01-17 05:44:07,128 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:44:07,130 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:44:07,683 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:44:08,274 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:44:08,298 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:44:08,666 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:44:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:44:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:40.738568+00:00 (in 29.999530 seconds) buildlogsarchiver stdout | 2023-01-17 05:44:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:40 UTC)" (scheduled at 2023-01-17 05:44:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:44:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 44, 10, 739421), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:44:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:44:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:44:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:40 UTC)" executed successfully builder stdout | 2023-01-17 05:44:13,560 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:13,573 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:13,573 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:13,584 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:25.646354+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 05:44:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:25 UTC)" (scheduled at 2023-01-17 05:44:15.646354+00:00) notificationworker stdout | 2023-01-17 05:44:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 15, 647278), True, datetime.datetime(2023, 1, 17, 5, 44, 15, 647278), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:19.537472+00:00 (in 2.600046 seconds) securityworker stdout | 2023-01-17 05:44:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:46 UTC)" (scheduled at 2023-01-17 05:44:16.936988+00:00) securityworker stdout | 2023-01-17 05:44:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:44:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:44:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:04.936835+00:00 (in 45.590224 seconds) repositorygcworker stdout | 2023-01-17 05:44:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:19 UTC)" (scheduled at 2023-01-17 05:44:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:44:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:44:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:46.936988+00:00 (in 27.399060 seconds) securityworker stdout | 2023-01-17 05:44:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:49 UTC)" (scheduled at 2023-01-17 05:44:19.537472+00:00) securityworker stdout | 2023-01-17 05:44:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:44:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:44:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:19,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:44:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:44:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:44:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:44:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:44:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:44:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:51.348605+00:00 (in 29.999583 seconds) gcworker stdout | 2023-01-17 05:44:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:51 UTC)" (scheduled at 2023-01-17 05:44:21.348605+00:00) gcworker stdout | 2023-01-17 05:44:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724661349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:44:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:44:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:44:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:44:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:44:22,060 [235] [DEBUG] [app] Starting request: urn:request:89b791c1-a5e7-4f75-943f-85524e0757aa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:22,060 [233] [DEBUG] [app] Starting request: urn:request:90827ee1-2123-4912-b193-521ea209fd23 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:22,065 [230] [DEBUG] [app] Starting request: urn:request:2ad574a3-49d5-47da-a565-c98fdc6867c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:22,066 [241] [DEBUG] [app] Starting request: urn:request:1c692434-5f38-4e15-948b-05c0829c8ba4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:22,066 [230] [DEBUG] [app] Ending request: urn:request:2ad574a3-49d5-47da-a565-c98fdc6867c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ad574a3-49d5-47da-a565-c98fdc6867c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:44:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:22,066 [241] [DEBUG] [app] Ending request: urn:request:1c692434-5f38-4e15-948b-05c0829c8ba4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c692434-5f38-4e15-948b-05c0829c8ba4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:44:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:44:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:22,070 [235] [DEBUG] [app] Starting request: urn:request:f6f5b1bd-fd89-45ae-b123-f7df9014daaf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:22,070 [235] [DEBUG] [app] Ending request: urn:request:f6f5b1bd-fd89-45ae-b123-f7df9014daaf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6f5b1bd-fd89-45ae-b123-f7df9014daaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:44:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:22,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:22,072 [231] [DEBUG] [app] Starting request: urn:request:dc451acc-4040-4c96-8934-a38c44009a2a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:22,072 [231] [DEBUG] [app] Ending request: urn:request:dc451acc-4040-4c96-8934-a38c44009a2a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dc451acc-4040-4c96-8934-a38c44009a2a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:22,073 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:44:22,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:22,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:22,087 [233] [DEBUG] [app] Ending request: urn:request:90827ee1-2123-4912-b193-521ea209fd23 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90827ee1-2123-4912-b193-521ea209fd23', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:44:22,090 [235] [DEBUG] [app] Ending request: urn:request:89b791c1-a5e7-4f75-943f-85524e0757aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:89b791c1-a5e7-4f75-943f-85524e0757aa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:22,091 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.031) gunicorn-web stdout | 2023-01-17 05:44:22,091 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:44:23,594 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:23,616 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:23,616 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:23,627 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:35.646354+00:00 (in 9.999492 seconds) notificationworker stdout | 2023-01-17 05:44:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:35 UTC)" (scheduled at 2023-01-17 05:44:25.646354+00:00) notificationworker stdout | 2023-01-17 05:44:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 25, 647342), True, datetime.datetime(2023, 1, 17, 5, 44, 25, 647342), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:44:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:44:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:35.853396+00:00 (in 6.889268 seconds) exportactionlogsworker stdout | 2023-01-17 05:44:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:28 UTC)" (scheduled at 2023-01-17 05:44:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:44:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:44:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:29,536 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:44:29,578 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:44:30,454 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:44:31,029 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:44:31,032 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:44:31,235 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:44:31,239 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:44:31,242 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:44:31,245 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:44:31,248 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:44:31,257 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:44:31,569 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:44:33,637 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:33,650 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:33,650 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:33,660 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:44:34,686 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:44:34,782 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:44:35,229 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:44:35,231 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:44:35,234 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:44:35,604 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:44:35,607 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:44:35,612 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:44:35,615 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:44:35,617 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:44:35,620 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:44:35,622 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:44:35,625 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:44:35,628 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} notificationworker stdout | 2023-01-17 05:44:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:45.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 05:44:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:45 UTC)" (scheduled at 2023-01-17 05:44:35.646354+00:00) notificationworker stdout | 2023-01-17 05:44:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 35, 647267), True, datetime.datetime(2023, 1, 17, 5, 44, 35, 647267), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:44:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:44:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:28.963662+00:00 (in 53.109762 seconds) exportactionlogsworker stdout | 2023-01-17 05:44:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:35 UTC)" (scheduled at 2023-01-17 05:44:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:44:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:44:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 35, 854348), True, datetime.datetime(2023, 1, 17, 5, 44, 35, 854348), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:44:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:44:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:44:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:44:35,924 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:44:35,940 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:44:36,174 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:44:36,685 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:44:36,892 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:44:37,060 [236] [DEBUG] [app] Starting request: urn:request:592b4a2e-aa28-4bbf-bc0f-53f2b7571e98 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:37,060 [231] [DEBUG] [app] Starting request: urn:request:4de94d37-f86b-45de-a6a3-3954acf07802 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:37,065 [239] [DEBUG] [app] Starting request: urn:request:9cb0854e-352f-461a-b8de-602cdc580275 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:37,065 [243] [DEBUG] [app] Starting request: urn:request:a684ff2d-da24-42f9-96b6-3d0c6c9487ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:37,065 [239] [DEBUG] [app] Ending request: urn:request:9cb0854e-352f-461a-b8de-602cdc580275 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9cb0854e-352f-461a-b8de-602cdc580275', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:44:37,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:44:37,066 [243] [DEBUG] [app] Ending request: urn:request:a684ff2d-da24-42f9-96b6-3d0c6c9487ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a684ff2d-da24-42f9-96b6-3d0c6c9487ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:44:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:44:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:37,070 [235] [DEBUG] [app] Starting request: urn:request:6d29ed0a-ad29-4857-b39d-ef6368f0fbb2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:37,070 [233] [DEBUG] [app] Starting request: urn:request:87be1d3a-ae69-44a7-bcae-f549c11f9eb2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:37,070 [235] [DEBUG] [app] Ending request: urn:request:6d29ed0a-ad29-4857-b39d-ef6368f0fbb2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d29ed0a-ad29-4857-b39d-ef6368f0fbb2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:37,071 [233] [DEBUG] [app] Ending request: urn:request:87be1d3a-ae69-44a7-bcae-f549c11f9eb2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87be1d3a-ae69-44a7-bcae-f549c11f9eb2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:44:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:37,088 [231] [DEBUG] [app] Ending request: urn:request:4de94d37-f86b-45de-a6a3-3954acf07802 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4de94d37-f86b-45de-a6a3-3954acf07802', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:37,088 [236] [DEBUG] [app] Ending request: urn:request:592b4a2e-aa28-4bbf-bc0f-53f2b7571e98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:592b4a2e-aa28-4bbf-bc0f-53f2b7571e98', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:44:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) repositoryactioncounter stdout | 2023-01-17 05:44:37,141 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:44:37,144 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:44:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:44:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:37.537407+00:00 (in 0.280395 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:44:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:37 UTC)" (scheduled at 2023-01-17 05:44:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:44:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:44:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 37, 257394), True, datetime.datetime(2023, 1, 17, 5, 44, 37, 257394), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:44:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:44:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:44:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:44:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:44:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:37.256652+00:00 (in 59.718884 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:44:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:37 UTC)" (scheduled at 2023-01-17 05:44:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:44:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:44:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:44:37,710 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:44:38,310 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:44:38,321 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:44:38,688 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:44:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:44:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:10.738568+00:00 (in 29.999523 seconds) buildlogsarchiver stdout | 2023-01-17 05:44:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:10 UTC)" (scheduled at 2023-01-17 05:44:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:44:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 44, 40, 739479), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:44:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:44:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:44:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:44:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:44:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:42.941536+00:00 (in 1.792832 seconds) namespacegcworker stdout | 2023-01-17 05:44:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:41 UTC)" (scheduled at 2023-01-17 05:44:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:44:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:44:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 41, 149109), True, datetime.datetime(2023, 1, 17, 5, 44, 41, 149109), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:44:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:44:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:44:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:44:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:44:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:41.148245+00:00 (in 58.206233 seconds) namespacegcworker stdout | 2023-01-17 05:44:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:42 UTC)" (scheduled at 2023-01-17 05:44:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:44:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:44:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:45:42 UTC)" executed successfully builder stdout | 2023-01-17 05:44:43,671 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:43,683 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:43,683 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:43,694 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:55.646354+00:00 (in 9.999392 seconds) notificationworker stdout | 2023-01-17 05:44:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:55 UTC)" (scheduled at 2023-01-17 05:44:45.646354+00:00) notificationworker stdout | 2023-01-17 05:44:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 45, 647390), True, datetime.datetime(2023, 1, 17, 5, 44, 45, 647390), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:44:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:44:49.537472+00:00 (in 2.599992 seconds) securityworker stdout | 2023-01-17 05:44:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:16 UTC)" (scheduled at 2023-01-17 05:44:46.936988+00:00) securityworker stdout | 2023-01-17 05:44:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:44:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:44:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:16.936988+00:00 (in 27.399052 seconds) securityworker stdout | 2023-01-17 05:44:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:19 UTC)" (scheduled at 2023-01-17 05:44:49.537472+00:00) securityworker stdout | 2023-01-17 05:44:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:44:49,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:49,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:44:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:44:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:44:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:44:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:44:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:44:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:44:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:44:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:44:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:44:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:44:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:44:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:44:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:44:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:21.348605+00:00 (in 29.999577 seconds) gcworker stdout | 2023-01-17 05:44:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:21 UTC)" (scheduled at 2023-01-17 05:44:51.348605+00:00) gcworker stdout | 2023-01-17 05:44:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724691349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:44:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:44:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:44:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:44:52,060 [235] [DEBUG] [app] Starting request: urn:request:55c2ec9b-800a-4b5f-b73f-8618af84f762 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:52,060 [231] [DEBUG] [app] Starting request: urn:request:90c72ce6-d471-4a78-a8a1-149118728a6d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:44:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:44:52,065 [230] [DEBUG] [app] Starting request: urn:request:07a7f57c-6cee-428f-b3a9-7ce79fac6e52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:52,066 [242] [DEBUG] [app] Starting request: urn:request:b2f60936-f4b8-4dc4-8593-f2f9545080ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:44:52,066 [230] [DEBUG] [app] Ending request: urn:request:07a7f57c-6cee-428f-b3a9-7ce79fac6e52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07a7f57c-6cee-428f-b3a9-7ce79fac6e52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:44:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:44:52,066 [242] [DEBUG] [app] Ending request: urn:request:b2f60936-f4b8-4dc4-8593-f2f9545080ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2f60936-f4b8-4dc4-8593-f2f9545080ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:52,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:44:52,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:52,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:44:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:44:52,072 [233] [DEBUG] [app] Starting request: urn:request:127dd555-054e-4a38-9c26-212622c84f8f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:52,072 [236] [DEBUG] [app] Starting request: urn:request:0f866370-bbd5-4b42-9ea2-c5c4c86b0513 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:44:52,072 [233] [DEBUG] [app] Ending request: urn:request:127dd555-054e-4a38-9c26-212622c84f8f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:127dd555-054e-4a38-9c26-212622c84f8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:44:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:52,072 [236] [DEBUG] [app] Ending request: urn:request:0f866370-bbd5-4b42-9ea2-c5c4c86b0513 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0f866370-bbd5-4b42-9ea2-c5c4c86b0513', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:44:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:44:52,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:44:52,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:44:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:52,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:44:52,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:52,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:44:52,089 [231] [DEBUG] [app] Ending request: urn:request:90c72ce6-d471-4a78-a8a1-149118728a6d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90c72ce6-d471-4a78-a8a1-149118728a6d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:44:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:44:52,089 [235] [DEBUG] [app] Ending request: urn:request:55c2ec9b-800a-4b5f-b73f-8618af84f762 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55c2ec9b-800a-4b5f-b73f-8618af84f762', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:44:52,089 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:44:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:44:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:44:53,704 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:44:53,718 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:44:53,718 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:44:53,728 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:44:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:01.636362+00:00 (in 5.989516 seconds) notificationworker stdout | 2023-01-17 05:44:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:05 UTC)" (scheduled at 2023-01-17 05:44:55.646354+00:00) notificationworker stdout | 2023-01-17 05:44:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:44:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 44, 55, 647277), True, datetime.datetime(2023, 1, 17, 5, 44, 55, 647277), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:44:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:44:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:44:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:44:59,567 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:44:59,613 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:45:00,484 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:45:01,049 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:45:01,064 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:45:01,243 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:45:01,246 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:45:01,248 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:45:01,251 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:45:01,254 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:45:01,292 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:45:01,604 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 05:45:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:05.646354+00:00 (in 4.009528 seconds) notificationworker stdout | 2023-01-17 05:45:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:01 UTC)" (scheduled at 2023-01-17 05:45:01.636362+00:00) notificationworker stdout | 2023-01-17 05:45:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:45:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:01 UTC)" executed successfully builder stdout | 2023-01-17 05:45:03,739 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:03,751 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:03,751 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:03,762 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:45:04,693 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:45:04,804 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:45:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:45:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:19.346120+00:00 (in 14.408847 seconds) repositorygcworker stdout | 2023-01-17 05:45:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:04 UTC)" (scheduled at 2023-01-17 05:45:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:45:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:45:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 4, 937702), True, datetime.datetime(2023, 1, 17, 5, 45, 4, 937702), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:45:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:45:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:45:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:45:05,236 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:45:05,239 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:45:05,241 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:45:05,615 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:45:05,618 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:45:05,620 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:45:05,623 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:45:05,627 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:45:05,629 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:45:05,632 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:45:05,635 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:45:05,637 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} notificationworker stdout | 2023-01-17 05:45:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:15.646354+00:00 (in 9.999559 seconds) notificationworker stdout | 2023-01-17 05:45:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:15 UTC)" (scheduled at 2023-01-17 05:45:05.646354+00:00) notificationworker stdout | 2023-01-17 05:45:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 5, 647150), True, datetime.datetime(2023, 1, 17, 5, 45, 5, 647150), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:15 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:45:05,934 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:45:05,974 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:45:06,182 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:45:06,706 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:45:06,910 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:45:07,060 [233] [DEBUG] [app] Starting request: urn:request:cd62accf-9ff5-445c-a410-30325808a6f6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:07,060 [235] [DEBUG] [app] Starting request: urn:request:be97ce91-d929-465e-aeca-14a3e578839b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:07,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:07,065 [241] [DEBUG] [app] Starting request: urn:request:62c67d97-6abf-46ed-a36c-9a9641fa4984 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:07,065 [230] [DEBUG] [app] Starting request: urn:request:f14eadf2-fd52-4c82-aa56-122fb2dd307e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:07,066 [241] [DEBUG] [app] Ending request: urn:request:62c67d97-6abf-46ed-a36c-9a9641fa4984 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:62c67d97-6abf-46ed-a36c-9a9641fa4984', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:45:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:45:07,066 [230] [DEBUG] [app] Ending request: urn:request:f14eadf2-fd52-4c82-aa56-122fb2dd307e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f14eadf2-fd52-4c82-aa56-122fb2dd307e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:45:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:45:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:07,071 [236] [DEBUG] [app] Starting request: urn:request:9a5d3b56-8a0c-4a88-885e-5368cd9b5d18 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:07,071 [233] [DEBUG] [app] Starting request: urn:request:c3aa2097-3a19-48a8-bc53-8d54e17b501c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:07,071 [236] [DEBUG] [app] Ending request: urn:request:9a5d3b56-8a0c-4a88-885e-5368cd9b5d18 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9a5d3b56-8a0c-4a88-885e-5368cd9b5d18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:07,071 [233] [DEBUG] [app] Ending request: urn:request:c3aa2097-3a19-48a8-bc53-8d54e17b501c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c3aa2097-3a19-48a8-bc53-8d54e17b501c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:07,088 [235] [DEBUG] [app] Ending request: urn:request:be97ce91-d929-465e-aeca-14a3e578839b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:be97ce91-d929-465e-aeca-14a3e578839b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:07,088 [233] [DEBUG] [app] Ending request: urn:request:cd62accf-9ff5-445c-a410-30325808a6f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd62accf-9ff5-445c-a410-30325808a6f6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:45:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:45:07,159 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:45:07,176 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:45:07,746 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:45:08,342 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:45:08,345 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:45:08,721 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:45:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:45:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:40.738568+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2023-01-17 05:45:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:40 UTC)" (scheduled at 2023-01-17 05:45:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:45:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 45, 10, 739421), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:45:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:45:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:45:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:40 UTC)" executed successfully builder stdout | 2023-01-17 05:45:13,765 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:13,777 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:13,777 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:13,788 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:25.646354+00:00 (in 9.999559 seconds) notificationworker stdout | 2023-01-17 05:45:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:25 UTC)" (scheduled at 2023-01-17 05:45:15.646354+00:00) notificationworker stdout | 2023-01-17 05:45:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 15, 647190), True, datetime.datetime(2023, 1, 17, 5, 45, 15, 647190), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:45:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:19.537472+00:00 (in 2.600015 seconds) securityworker stdout | 2023-01-17 05:45:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:46 UTC)" (scheduled at 2023-01-17 05:45:16.936988+00:00) securityworker stdout | 2023-01-17 05:45:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:45:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:45:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:04.936835+00:00 (in 45.590266 seconds) repositorygcworker stdout | 2023-01-17 05:45:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:19 UTC)" (scheduled at 2023-01-17 05:45:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:45:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:45:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:45:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:46.936988+00:00 (in 27.399049 seconds) securityworker stdout | 2023-01-17 05:45:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:49 UTC)" (scheduled at 2023-01-17 05:45:19.537472+00:00) securityworker stdout | 2023-01-17 05:45:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:45:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:45:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:45:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:45:19,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:45:19,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:45:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:45:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:45:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:51.348605+00:00 (in 29.999548 seconds) gcworker stdout | 2023-01-17 05:45:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:51 UTC)" (scheduled at 2023-01-17 05:45:21.348605+00:00) gcworker stdout | 2023-01-17 05:45:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724721349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:45:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:45:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:45:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:45:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:45:22,060 [231] [DEBUG] [app] Starting request: urn:request:dad60954-6da7-4da0-afdd-52ddb57c06c8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:22,060 [233] [DEBUG] [app] Starting request: urn:request:2d6520e9-989c-4bba-949c-e61939c2ea4f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:22,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:22,066 [243] [DEBUG] [app] Starting request: urn:request:07b25ad0-816e-4c18-b384-7e0697564421 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:22,066 [242] [DEBUG] [app] Starting request: urn:request:a0e6c138-972e-408a-a257-dc0ad6abf3ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:22,066 [243] [DEBUG] [app] Ending request: urn:request:07b25ad0-816e-4c18-b384-7e0697564421 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07b25ad0-816e-4c18-b384-7e0697564421', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:45:22,066 [242] [DEBUG] [app] Ending request: urn:request:a0e6c138-972e-408a-a257-dc0ad6abf3ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0e6c138-972e-408a-a257-dc0ad6abf3ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:45:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:45:22,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:45:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:22,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:22,071 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:22,071 [236] [DEBUG] [app] Starting request: urn:request:fc4caecf-84b1-478b-b820-ea5733c4365e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:22,071 [231] [DEBUG] [app] Starting request: urn:request:aef1cdb9-681b-40d0-8435-95e939a406ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:22,072 [236] [DEBUG] [app] Ending request: urn:request:fc4caecf-84b1-478b-b820-ea5733c4365e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc4caecf-84b1-478b-b820-ea5733c4365e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:22,072 [231] [DEBUG] [app] Ending request: urn:request:aef1cdb9-681b-40d0-8435-95e939a406ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aef1cdb9-681b-40d0-8435-95e939a406ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:22,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:22,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:22,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:22,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:22,089 [233] [DEBUG] [app] Ending request: urn:request:2d6520e9-989c-4bba-949c-e61939c2ea4f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d6520e9-989c-4bba-949c-e61939c2ea4f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:22,089 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:22,090 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:45:22,096 [231] [DEBUG] [app] Ending request: urn:request:dad60954-6da7-4da0-afdd-52ddb57c06c8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dad60954-6da7-4da0-afdd-52ddb57c06c8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:22,096 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:22,096 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.037) builder stdout | 2023-01-17 05:45:23,793 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:23,805 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:23,805 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:23,816 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:35.646354+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 05:45:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:35 UTC)" (scheduled at 2023-01-17 05:45:25.646354+00:00) notificationworker stdout | 2023-01-17 05:45:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 25, 647229), True, datetime.datetime(2023, 1, 17, 5, 45, 25, 647229), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:45:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:45:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:35.853396+00:00 (in 6.889267 seconds) exportactionlogsworker stdout | 2023-01-17 05:45:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:28 UTC)" (scheduled at 2023-01-17 05:45:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:45:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:45:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:29,591 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:45:29,648 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:45:30,515 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:45:31,085 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:45:31,099 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:45:31,251 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:45:31,254 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:45:31,257 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:45:31,259 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:45:31,261 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:45:31,299 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:45:31,639 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:45:33,826 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:33,839 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:33,839 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:33,849 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:45:34,699 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:45:34,826 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:45:35,243 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:45:35,246 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:45:35,248 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:45:35,626 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:45:35,629 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:45:35,631 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:45:35,634 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:45:35,637 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:45:35,639 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:45:35,641 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:45:35,644 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:45:35,646 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} notificationworker stdout | 2023-01-17 05:45:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:45.646354+00:00 (in 9.999564 seconds) notificationworker stdout | 2023-01-17 05:45:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:45 UTC)" (scheduled at 2023-01-17 05:45:35.646354+00:00) notificationworker stdout | 2023-01-17 05:45:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 35, 647155), True, datetime.datetime(2023, 1, 17, 5, 45, 35, 647155), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:45:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:45:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:28.963662+00:00 (in 53.109749 seconds) exportactionlogsworker stdout | 2023-01-17 05:45:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:35 UTC)" (scheduled at 2023-01-17 05:45:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:45:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:45:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 35, 854334), True, datetime.datetime(2023, 1, 17, 5, 45, 35, 854334), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:45:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:45:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:45:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:45:35,958 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:45:36,007 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:45:36,215 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:45:36,741 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:45:36,944 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:45:37,059 [235] [DEBUG] [app] Starting request: urn:request:f95cee37-527b-4842-9dc2-1e2f15a93bac (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:37,059 [231] [DEBUG] [app] Starting request: urn:request:7722e644-9f20-4bf3-8423-be37968c2bf8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:37,064 [230] [DEBUG] [app] Starting request: urn:request:f3f59802-c447-4ff6-a1ae-5eba2968df03 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:37,064 [230] [DEBUG] [app] Ending request: urn:request:f3f59802-c447-4ff6-a1ae-5eba2968df03 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3f59802-c447-4ff6-a1ae-5eba2968df03', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:45:37,065 [243] [DEBUG] [app] Starting request: urn:request:802408fa-88e4-408d-99d4-ec044df008e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:45:37,065 [243] [DEBUG] [app] Ending request: urn:request:802408fa-88e4-408d-99d4-ec044df008e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:802408fa-88e4-408d-99d4-ec044df008e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:45:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:45:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:37,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:37,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:37,069 [236] [DEBUG] [app] Starting request: urn:request:195bfd1a-de52-4f6f-9d51-e88e7e179af9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:37,069 [236] [DEBUG] [app] Ending request: urn:request:195bfd1a-de52-4f6f-9d51-e88e7e179af9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:195bfd1a-de52-4f6f-9d51-e88e7e179af9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:37,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:37,071 [233] [DEBUG] [app] Starting request: urn:request:05edc86d-97cf-47a1-bcd0-dc987983398a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:37,071 [233] [DEBUG] [app] Ending request: urn:request:05edc86d-97cf-47a1-bcd0-dc987983398a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05edc86d-97cf-47a1-bcd0-dc987983398a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:37,087 [231] [DEBUG] [app] Ending request: urn:request:7722e644-9f20-4bf3-8423-be37968c2bf8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7722e644-9f20-4bf3-8423-be37968c2bf8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:45:37,088 [235] [DEBUG] [app] Ending request: urn:request:f95cee37-527b-4842-9dc2-1e2f15a93bac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f95cee37-527b-4842-9dc2-1e2f15a93bac', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:45:37,165 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:45:37,181 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:45:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:45:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:37.537407+00:00 (in 0.280366 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:45:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:37 UTC)" (scheduled at 2023-01-17 05:45:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:45:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:45:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 37, 257469), True, datetime.datetime(2023, 1, 17, 5, 45, 37, 257469), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:45:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:45:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:45:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:45:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:45:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:23.537048+00:00 (in 45.999224 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:45:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:37 UTC)" (scheduled at 2023-01-17 05:45:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:45:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:45:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:45:37,782 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:45:38,358 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:45:38,379 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:45:38,731 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:45:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:45:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:10.738568+00:00 (in 29.999566 seconds) buildlogsarchiver stdout | 2023-01-17 05:45:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:10 UTC)" (scheduled at 2023-01-17 05:45:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:45:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 45, 40, 739408), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:45:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:45:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:45:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:45:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:45:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:42.941536+00:00 (in 1.792822 seconds) namespacegcworker stdout | 2023-01-17 05:45:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:41 UTC)" (scheduled at 2023-01-17 05:45:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:45:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:45:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 41, 149113), True, datetime.datetime(2023, 1, 17, 5, 45, 41, 149113), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:45:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:45:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:45:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:45:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:45:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:41.148245+00:00 (in 58.206258 seconds) namespacegcworker stdout | 2023-01-17 05:45:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:42 UTC)" (scheduled at 2023-01-17 05:45:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:45:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:45:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:46:42 UTC)" executed successfully builder stdout | 2023-01-17 05:45:43,860 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:43,872 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:43,872 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:43,883 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:55.646354+00:00 (in 9.999572 seconds) notificationworker stdout | 2023-01-17 05:45:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:55 UTC)" (scheduled at 2023-01-17 05:45:45.646354+00:00) notificationworker stdout | 2023-01-17 05:45:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 45, 647200), True, datetime.datetime(2023, 1, 17, 5, 45, 45, 647200), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:45:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:45:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:45:49.537472+00:00 (in 2.600028 seconds) securityworker stdout | 2023-01-17 05:45:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:16 UTC)" (scheduled at 2023-01-17 05:45:46.936988+00:00) securityworker stdout | 2023-01-17 05:45:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:45:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:45:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:16.936988+00:00 (in 27.399045 seconds) securityworker stdout | 2023-01-17 05:45:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:19 UTC)" (scheduled at 2023-01-17 05:45:49.537472+00:00) securityworker stdout | 2023-01-17 05:45:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:45:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:45:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:45:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:45:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:45:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:45:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:45:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:45:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:45:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:45:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:45:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:45:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:45:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:45:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:45:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:21.348605+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 05:45:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:21 UTC)" (scheduled at 2023-01-17 05:45:51.348605+00:00) gcworker stdout | 2023-01-17 05:45:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724751349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:45:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:45:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:45:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:45:52,059 [231] [DEBUG] [app] Starting request: urn:request:42dec290-9e99-4888-b939-9e67d354cc79 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:52,059 [235] [DEBUG] [app] Starting request: urn:request:5bf3a7aa-ca7e-40f1-bc7c-a7eae976f883 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:45:52,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,060 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:45:52,064 [243] [DEBUG] [app] Starting request: urn:request:11f9d403-5aa6-47ab-9fbd-3e874fc5a04d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:52,064 [242] [DEBUG] [app] Starting request: urn:request:77204047-f6a7-482d-afe9-887f493bd00c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:45:52,065 [243] [DEBUG] [app] Ending request: urn:request:11f9d403-5aa6-47ab-9fbd-3e874fc5a04d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11f9d403-5aa6-47ab-9fbd-3e874fc5a04d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:45:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:45:52,065 [242] [DEBUG] [app] Ending request: urn:request:77204047-f6a7-482d-afe9-887f493bd00c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77204047-f6a7-482d-afe9-887f493bd00c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:52,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:45:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:45:52,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:45:52,070 [236] [DEBUG] [app] Starting request: urn:request:d9a363d4-f2aa-4e4a-9bd2-a22d56d1eab4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:52,070 [233] [DEBUG] [app] Starting request: urn:request:e0fd5d8a-42af-4b11-a5d6-859c3f71f765 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:45:52,070 [236] [DEBUG] [app] Ending request: urn:request:d9a363d4-f2aa-4e4a-9bd2-a22d56d1eab4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9a363d4-f2aa-4e4a-9bd2-a22d56d1eab4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:52,071 [233] [DEBUG] [app] Ending request: urn:request:e0fd5d8a-42af-4b11-a5d6-859c3f71f765 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0fd5d8a-42af-4b11-a5d6-859c3f71f765', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:45:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:45:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:45:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:45:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:45:52,085 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:52,086 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:45:52,094 [231] [DEBUG] [app] Ending request: urn:request:42dec290-9e99-4888-b939-9e67d354cc79 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42dec290-9e99-4888-b939-9e67d354cc79', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:52,094 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:45:52,095 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.036) gunicorn-web stdout | 2023-01-17 05:45:52,095 [235] [DEBUG] [app] Ending request: urn:request:5bf3a7aa-ca7e-40f1-bc7c-a7eae976f883 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5bf3a7aa-ca7e-40f1-bc7c-a7eae976f883', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:45:52,095 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.038 47 0.037) gunicorn-web stdout | 2023-01-17 05:45:52,096 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:45:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:45:53,884 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:45:53,896 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:45:53,896 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:45:53,907 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:45:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:01.636362+00:00 (in 5.989539 seconds) notificationworker stdout | 2023-01-17 05:45:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:05 UTC)" (scheduled at 2023-01-17 05:45:55.646354+00:00) notificationworker stdout | 2023-01-17 05:45:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:45:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 45, 55, 647240), True, datetime.datetime(2023, 1, 17, 5, 45, 55, 647240), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:45:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:45:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:45:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:45:59,627 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:45:59,663 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:46:00,535 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} builder stdout | 2023-01-17 05:46:01,106 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:46:01,117 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:46:01,260 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:46:01,262 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:46:01,265 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:46:01,267 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:46:01,269 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:46:01,321 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:46:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:05.646354+00:00 (in 4.009516 seconds) notificationworker stdout | 2023-01-17 05:46:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:01 UTC)" (scheduled at 2023-01-17 05:46:01.636362+00:00) notificationworker stdout | 2023-01-17 05:46:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:46:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:01,675 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:46:03,917 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:03,929 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:03,929 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:03,940 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:46:04,734 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:46:04,841 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:46:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:46:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:17.345601+00:00 (in 12.408332 seconds) repositorygcworker stdout | 2023-01-17 05:46:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:04 UTC)" (scheduled at 2023-01-17 05:46:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:46:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:46:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 4, 937698), True, datetime.datetime(2023, 1, 17, 5, 46, 4, 937698), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:46:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:46:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:46:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:46:05,251 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:46:05,254 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:46:05,256 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 05:46:05,637 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:46:05,640 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:46:05,643 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:46:05,645 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} notificationworker stdout | 2023-01-17 05:46:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:15.646354+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 05:46:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:15 UTC)" (scheduled at 2023-01-17 05:46:05.646354+00:00) notificationworker stdout | 2023-01-17 05:46:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. gunicorn-registry stdout | 2023-01-17 05:46:05,647 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} notificationworker stdout | 2023-01-17 05:46:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 5, 647159), True, datetime.datetime(2023, 1, 17, 5, 46, 5, 647159), 0, 'notification/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:46:05,650 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:46:05,652 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:46:05,655 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:46:05,657 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} notificationworker stdout | 2023-01-17 05:46:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:15 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:46:05,993 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:46:06,024 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:46:06,249 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:46:06,777 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:46:06,979 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:46:07,060 [235] [DEBUG] [app] Starting request: urn:request:b69d9a0f-3854-485e-bef5-17ec7606ae1d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:07,060 [233] [DEBUG] [app] Starting request: urn:request:2675e1cc-cc15-4244-8be2-282a5324c01a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:07,065 [240] [DEBUG] [app] Starting request: urn:request:95874ae6-12cd-47b9-9c13-a250a231bbcf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:07,065 [230] [DEBUG] [app] Starting request: urn:request:14f1ac2f-9ee6-4a38-946d-dcc2de4cee71 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:07,065 [240] [DEBUG] [app] Ending request: urn:request:95874ae6-12cd-47b9-9c13-a250a231bbcf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:95874ae6-12cd-47b9-9c13-a250a231bbcf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:46:07,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:46:07,066 [230] [DEBUG] [app] Ending request: urn:request:14f1ac2f-9ee6-4a38-946d-dcc2de4cee71 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:14f1ac2f-9ee6-4a38-946d-dcc2de4cee71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:46:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:46:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:07,070 [231] [DEBUG] [app] Starting request: urn:request:72ed8775-4b2e-4e9e-98ce-dfb5b07733df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:07,070 [236] [DEBUG] [app] Starting request: urn:request:cb6362e0-226a-470f-aa96-86b2c98fb12f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:07,070 [231] [DEBUG] [app] Ending request: urn:request:72ed8775-4b2e-4e9e-98ce-dfb5b07733df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:72ed8775-4b2e-4e9e-98ce-dfb5b07733df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:07,071 [236] [DEBUG] [app] Ending request: urn:request:cb6362e0-226a-470f-aa96-86b2c98fb12f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cb6362e0-226a-470f-aa96-86b2c98fb12f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:46:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:07,088 [233] [DEBUG] [app] Ending request: urn:request:2675e1cc-cc15-4244-8be2-282a5324c01a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2675e1cc-cc15-4244-8be2-282a5324c01a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:07,088 [235] [DEBUG] [app] Ending request: urn:request:b69d9a0f-3854-485e-bef5-17ec7606ae1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b69d9a0f-3854-485e-bef5-17ec7606ae1d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:46:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) repositoryactioncounter stdout | 2023-01-17 05:46:07,190 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:46:07,200 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:46:07,801 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:46:08,393 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:46:08,398 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:46:08,767 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:46:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:46:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:40.738568+00:00 (in 29.999568 seconds) buildlogsarchiver stdout | 2023-01-17 05:46:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:40 UTC)" (scheduled at 2023-01-17 05:46:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:46:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 46, 10, 739359), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:46:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:46:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:46:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:40 UTC)" executed successfully builder stdout | 2023-01-17 05:46:13,950 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:13,963 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:13,963 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:13,974 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:25.646354+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:46:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:25 UTC)" (scheduled at 2023-01-17 05:46:15.646354+00:00) notificationworker stdout | 2023-01-17 05:46:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 15, 647229), True, datetime.datetime(2023, 1, 17, 5, 46, 15, 647229), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:19.537472+00:00 (in 2.600029 seconds) securityworker stdout | 2023-01-17 05:46:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:46 UTC)" (scheduled at 2023-01-17 05:46:16.936988+00:00) securityworker stdout | 2023-01-17 05:46:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:46:17,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:46:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:19.346120+00:00 (in 2.000005 seconds) repositorygcworker stdout | 2023-01-17 05:46:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:17 UTC)" (scheduled at 2023-01-17 05:46:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:46:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 46, 17, 346511), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:46:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 17, 346511), True, datetime.datetime(2023, 1, 17, 5, 46, 17, 346511), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:46:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 17, 346511), True, datetime.datetime(2023, 1, 17, 5, 46, 17, 346511), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 46, 17, 346511), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:46:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:46:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:46:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:46:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:04.936835+00:00 (in 45.590266 seconds) repositorygcworker stdout | 2023-01-17 05:46:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:19 UTC)" (scheduled at 2023-01-17 05:46:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:46:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:46:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:46.936988+00:00 (in 27.399043 seconds) securityworker stdout | 2023-01-17 05:46:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:49 UTC)" (scheduled at 2023-01-17 05:46:19.537472+00:00) securityworker stdout | 2023-01-17 05:46:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:46:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:46:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:46:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:46:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:46:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:46:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:46:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:46:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:51.348605+00:00 (in 29.999531 seconds) gcworker stdout | 2023-01-17 05:46:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:51 UTC)" (scheduled at 2023-01-17 05:46:21.348605+00:00) gcworker stdout | 2023-01-17 05:46:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724781349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:46:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:46:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:46:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:46:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:46:22,060 [231] [DEBUG] [app] Starting request: urn:request:2cc62ba6-a518-46f0-97fa-76847188e94e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:22,060 [236] [DEBUG] [app] Starting request: urn:request:e0219eda-1665-48f3-bb72-903075ba1186 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:22,065 [230] [DEBUG] [app] Starting request: urn:request:06304261-a96c-4d69-b948-ed9e1ec33124 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:22,066 [241] [DEBUG] [app] Starting request: urn:request:db5e262a-b600-4ee4-9ded-a9f800855989 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:22,066 [230] [DEBUG] [app] Ending request: urn:request:06304261-a96c-4d69-b948-ed9e1ec33124 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:06304261-a96c-4d69-b948-ed9e1ec33124', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:46:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:46:22,066 [241] [DEBUG] [app] Ending request: urn:request:db5e262a-b600-4ee4-9ded-a9f800855989 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:db5e262a-b600-4ee4-9ded-a9f800855989', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:46:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:46:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:22,070 [233] [DEBUG] [app] Starting request: urn:request:449a708c-5f15-414b-aee2-6af159bdd4fe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:22,071 [235] [DEBUG] [app] Starting request: urn:request:75eafdd5-43dd-4853-85f8-1f242c4ebb26 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:22,071 [233] [DEBUG] [app] Ending request: urn:request:449a708c-5f15-414b-aee2-6af159bdd4fe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:449a708c-5f15-414b-aee2-6af159bdd4fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:22,071 [235] [DEBUG] [app] Ending request: urn:request:75eafdd5-43dd-4853-85f8-1f242c4ebb26 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:75eafdd5-43dd-4853-85f8-1f242c4ebb26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:46:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:22,088 [236] [DEBUG] [app] Ending request: urn:request:e0219eda-1665-48f3-bb72-903075ba1186 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0219eda-1665-48f3-bb72-903075ba1186', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:22,088 [231] [DEBUG] [app] Ending request: urn:request:2cc62ba6-a518-46f0-97fa-76847188e94e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2cc62ba6-a518-46f0-97fa-76847188e94e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:46:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:46:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:46:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:37.256652+00:00 (in 13.719196 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:46:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:23 UTC)" (scheduled at 2023-01-17 05:46:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:46:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 46, 23, 537815), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:46:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 23, 537815), True, datetime.datetime(2023, 1, 17, 5, 46, 23, 537815), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:46:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 23, 537815), True, datetime.datetime(2023, 1, 17, 5, 46, 23, 537815), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 46, 23, 537815), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:46:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:46:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:23 UTC)" executed successfully builder stdout | 2023-01-17 05:46:23,983 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:23,995 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:23,995 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:24,006 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:35.646354+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 05:46:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:35 UTC)" (scheduled at 2023-01-17 05:46:25.646354+00:00) notificationworker stdout | 2023-01-17 05:46:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 25, 647225), True, datetime.datetime(2023, 1, 17, 5, 46, 25, 647225), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:25,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:25,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:25,662 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:46:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:46:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:35.853396+00:00 (in 6.889227 seconds) exportactionlogsworker stdout | 2023-01-17 05:46:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:28 UTC)" (scheduled at 2023-01-17 05:46:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:46:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:46:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:29,637 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:46:29,697 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:46:30,563 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:46:31,141 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:46:31,144 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:46:31,267 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:46:31,269 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:46:31,272 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:46:31,275 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:46:31,277 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:46:31,331 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:46:31,711 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:46:34,016 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:34,029 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:34,029 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:34,039 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:46:34,770 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:46:34,847 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:46:35,259 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:46:35,261 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:46:35,264 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:46:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:45.646354+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 05:46:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:45 UTC)" (scheduled at 2023-01-17 05:46:35.646354+00:00) notificationworker stdout | 2023-01-17 05:46:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. gunicorn-registry stdout | 2023-01-17 05:46:35,647 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} notificationworker stdout | 2023-01-17 05:46:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 35, 647174), True, datetime.datetime(2023, 1, 17, 5, 46, 35, 647174), 0, 'notification/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:46:35,649 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:46:35,652 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:46:35,654 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:46:35,657 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} notificationworker stdout | 2023-01-17 05:46:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:46:35,660 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:46:35,662 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:46:35,665 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:46:35,667 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:46:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:46:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:54.963413+00:00 (in 19.109403 seconds) exportactionlogsworker stdout | 2023-01-17 05:46:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:35 UTC)" (scheduled at 2023-01-17 05:46:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:46:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:46:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 35, 854484), True, datetime.datetime(2023, 1, 17, 5, 46, 35, 854484), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:46:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:46:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:46:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:46:35,998 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:46:36,048 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:46:36,257 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:46:36,813 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:46:37,015 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:46:37,059 [236] [DEBUG] [app] Starting request: urn:request:d86bf1df-731a-4589-bd89-739b0b9dd60e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:37,060 [231] [DEBUG] [app] Starting request: urn:request:db2c69bb-7191-4d4a-a88d-7bd34cf4edbb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:37,064 [242] [DEBUG] [app] Starting request: urn:request:79cfb5fe-8da8-48ce-8944-03a42d364d52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:37,065 [230] [DEBUG] [app] Starting request: urn:request:57ff718a-0642-46b7-b22f-7a647b062bda (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:37,065 [242] [DEBUG] [app] Ending request: urn:request:79cfb5fe-8da8-48ce-8944-03a42d364d52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:79cfb5fe-8da8-48ce-8944-03a42d364d52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:46:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:46:37,065 [230] [DEBUG] [app] Ending request: urn:request:57ff718a-0642-46b7-b22f-7a647b062bda (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57ff718a-0642-46b7-b22f-7a647b062bda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:46:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:46:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:46:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:37,070 [235] [DEBUG] [app] Starting request: urn:request:7a8ad15d-b66f-43fb-81f9-1d8bf112ad14 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:37,070 [231] [DEBUG] [app] Starting request: urn:request:284498d6-e3eb-43f0-90fc-6e31beecedfe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:37,071 [235] [DEBUG] [app] Ending request: urn:request:7a8ad15d-b66f-43fb-81f9-1d8bf112ad14 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a8ad15d-b66f-43fb-81f9-1d8bf112ad14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:37,071 [231] [DEBUG] [app] Ending request: urn:request:284498d6-e3eb-43f0-90fc-6e31beecedfe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:284498d6-e3eb-43f0-90fc-6e31beecedfe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:37,088 [231] [DEBUG] [app] Ending request: urn:request:db2c69bb-7191-4d4a-a88d-7bd34cf4edbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db2c69bb-7191-4d4a-a88d-7bd34cf4edbb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:37,088 [236] [DEBUG] [app] Ending request: urn:request:d86bf1df-731a-4589-bd89-739b0b9dd60e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d86bf1df-731a-4589-bd89-739b0b9dd60e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:46:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) repositoryactioncounter stdout | 2023-01-17 05:46:37,201 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:46:37,223 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:46:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:46:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:37.537407+00:00 (in 0.280366 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:46:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:37 UTC)" (scheduled at 2023-01-17 05:46:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:46:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:46:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 37, 257423), True, datetime.datetime(2023, 1, 17, 5, 46, 37, 257423), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:46:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:46:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:46:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:46:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:46:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:37.256652+00:00 (in 59.718861 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:46:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:37 UTC)" (scheduled at 2023-01-17 05:46:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:46:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:46:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:46:37,838 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:46:38,423 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:46:38,432 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:46:38,802 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:46:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:46:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:10.738568+00:00 (in 29.999540 seconds) buildlogsarchiver stdout | 2023-01-17 05:46:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:10 UTC)" (scheduled at 2023-01-17 05:46:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:46:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 46, 40, 739435), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:46:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:46:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:46:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:46:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:46:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:42.941536+00:00 (in 1.792825 seconds) namespacegcworker stdout | 2023-01-17 05:46:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:41 UTC)" (scheduled at 2023-01-17 05:46:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:46:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:46:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 41, 149103), True, datetime.datetime(2023, 1, 17, 5, 46, 41, 149103), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:46:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:46:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:46:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:46:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:46:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:06.941185+00:00 (in 23.999182 seconds) namespacegcworker stdout | 2023-01-17 05:46:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:42 UTC)" (scheduled at 2023-01-17 05:46:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:46:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:46:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:47:42 UTC)" executed successfully builder stdout | 2023-01-17 05:46:44,049 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:44,061 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:44,061 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:44,072 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:55.646354+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 05:46:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:55 UTC)" (scheduled at 2023-01-17 05:46:45.646354+00:00) notificationworker stdout | 2023-01-17 05:46:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 45, 647256), True, datetime.datetime(2023, 1, 17, 5, 46, 45, 647256), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:45,662 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:45,662 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:45,662 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:46:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:46:49.537472+00:00 (in 2.600037 seconds) securityworker stdout | 2023-01-17 05:46:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:16 UTC)" (scheduled at 2023-01-17 05:46:46.936988+00:00) securityworker stdout | 2023-01-17 05:46:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:46:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:46:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:16.936988+00:00 (in 27.399096 seconds) securityworker stdout | 2023-01-17 05:46:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:19 UTC)" (scheduled at 2023-01-17 05:46:49.537472+00:00) securityworker stdout | 2023-01-17 05:46:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:46:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:46:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:46:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:46:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:46:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:46:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:46:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:46:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:46:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:46:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:46:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:46:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:46:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:46:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:46:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:21.348605+00:00 (in 29.999544 seconds) gcworker stdout | 2023-01-17 05:46:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:21 UTC)" (scheduled at 2023-01-17 05:46:51.348605+00:00) gcworker stdout | 2023-01-17 05:46:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724811349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:46:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:46:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:46:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:46:52,060 [231] [DEBUG] [app] Starting request: urn:request:07a09368-863b-4d20-ab05-f8ce0691ba93 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:52,060 [236] [DEBUG] [app] Starting request: urn:request:fd232ad9-788a-459e-ac31-0026272619fe (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:46:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:46:52,064 [242] [DEBUG] [app] Starting request: urn:request:54b5358d-8a58-4a93-9b6d-6c2260deaacd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:52,065 [230] [DEBUG] [app] Starting request: urn:request:dbc92a2c-2f4f-4273-b2d3-1dd8b32feaa4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:46:52,065 [242] [DEBUG] [app] Ending request: urn:request:54b5358d-8a58-4a93-9b6d-6c2260deaacd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54b5358d-8a58-4a93-9b6d-6c2260deaacd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:46:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:46:52,065 [230] [DEBUG] [app] Ending request: urn:request:dbc92a2c-2f4f-4273-b2d3-1dd8b32feaa4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dbc92a2c-2f4f-4273-b2d3-1dd8b32feaa4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:46:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:46:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:46:52,070 [235] [DEBUG] [app] Starting request: urn:request:c3c39068-9755-4e42-b0c9-22c0c01b19cd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:52,070 [231] [DEBUG] [app] Starting request: urn:request:97a0607a-3aa5-4fb5-8db4-a488d0ec9e31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:46:52,070 [235] [DEBUG] [app] Ending request: urn:request:c3c39068-9755-4e42-b0c9-22c0c01b19cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c3c39068-9755-4e42-b0c9-22c0c01b19cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:46:52,070 [231] [DEBUG] [app] Ending request: urn:request:97a0607a-3aa5-4fb5-8db4-a488d0ec9e31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97a0607a-3aa5-4fb5-8db4-a488d0ec9e31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:46:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:46:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:46:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:46:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:52,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:46:52,087 [231] [DEBUG] [app] Ending request: urn:request:07a09368-863b-4d20-ab05-f8ce0691ba93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07a09368-863b-4d20-ab05-f8ce0691ba93', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:46:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:46:52,089 [236] [DEBUG] [app] Ending request: urn:request:fd232ad9-788a-459e-ac31-0026272619fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd232ad9-788a-459e-ac31-0026272619fe', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:46:52,089 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-web stdout | 2023-01-17 05:46:52,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:46:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:46:54,078 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:46:54,095 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:46:54,095 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:46:54,106 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:46:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:46:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:28.963662+00:00 (in 33.999739 seconds) exportactionlogsworker stdout | 2023-01-17 05:46:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:54 UTC)" (scheduled at 2023-01-17 05:46:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:46:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 46, 54, 964433), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:46:54,976 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 54, 964433), True, datetime.datetime(2023, 1, 17, 5, 46, 54, 964433), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:46:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 46, 54, 964433), True, datetime.datetime(2023, 1, 17, 5, 46, 54, 964433), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 46, 54, 964433), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:46:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:46:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:51:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:46:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:01.636362+00:00 (in 5.989519 seconds) notificationworker stdout | 2023-01-17 05:46:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:05 UTC)" (scheduled at 2023-01-17 05:46:55.646354+00:00) notificationworker stdout | 2023-01-17 05:46:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:46:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 46, 55, 647260), True, datetime.datetime(2023, 1, 17, 5, 46, 55, 647260), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:46:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:46:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:46:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:46:59,673 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:46:59,705 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:47:00,591 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:47:01,177 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:47:01,180 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:47:01,273 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:47:01,276 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:47:01,279 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:47:01,282 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:47:01,284 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:47:01,356 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:47:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:05.646354+00:00 (in 4.009528 seconds) notificationworker stdout | 2023-01-17 05:47:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:01 UTC)" (scheduled at 2023-01-17 05:47:01.636362+00:00) notificationworker stdout | 2023-01-17 05:47:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:47:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:01,747 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:47:04,108 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:04,120 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:04,121 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:04,131 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:47:04,799 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:47:04,878 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:47:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:47:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:19.346120+00:00 (in 14.408861 seconds) repositorygcworker stdout | 2023-01-17 05:47:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:04 UTC)" (scheduled at 2023-01-17 05:47:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:47:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:47:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 4, 937707), True, datetime.datetime(2023, 1, 17, 5, 47, 4, 937707), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:47:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:47:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:47:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:47:05,267 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:47:05,269 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:47:05,272 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:47:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:15.646354+00:00 (in 9.999503 seconds) notificationworker stdout | 2023-01-17 05:47:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:15 UTC)" (scheduled at 2023-01-17 05:47:05.646354+00:00) notificationworker stdout | 2023-01-17 05:47:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 5, 647231), True, datetime.datetime(2023, 1, 17, 5, 47, 5, 647231), 0, 'notification/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:47:05,658 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} notificationworker stdout | 2023-01-17 05:47:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:47:05,660 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:47:05,662 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:47:05,665 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:47:05,668 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:47:05,670 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:47:05,672 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:47:05,675 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:47:05,677 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:47:06,007 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:47:06,070 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:47:06,284 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:47:06,848 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} namespacegcworker stdout | 2023-01-17 05:47:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:47:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:41.148245+00:00 (in 34.206574 seconds) namespacegcworker stdout | 2023-01-17 05:47:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:52:06 UTC)" (scheduled at 2023-01-17 05:47:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:47:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 47, 6, 942074), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:47:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 47, 6, 942074), True, datetime.datetime(2023, 1, 17, 5, 47, 6, 942074), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:47:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 47, 6, 942074), True, datetime.datetime(2023, 1, 17, 5, 47, 6, 942074), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 47, 6, 942074), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:47:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:47:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:52:06 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:47:07,050 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:47:07,060 [236] [DEBUG] [app] Starting request: urn:request:de6165dc-b3d7-4d68-ad6c-bcc0f899a8c5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:07,060 [235] [DEBUG] [app] Starting request: urn:request:6d26bb9d-82cd-47d8-9750-5529a62058d2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:07,065 [230] [DEBUG] [app] Starting request: urn:request:f09dc9af-0ab3-4e88-b793-e0da28c2b63a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:07,065 [242] [DEBUG] [app] Starting request: urn:request:5490158b-0c9e-497a-b5f1-136a6556956a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:07,065 [230] [DEBUG] [app] Ending request: urn:request:f09dc9af-0ab3-4e88-b793-e0da28c2b63a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f09dc9af-0ab3-4e88-b793-e0da28c2b63a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:07,065 [242] [DEBUG] [app] Ending request: urn:request:5490158b-0c9e-497a-b5f1-136a6556956a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5490158b-0c9e-497a-b5f1-136a6556956a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:47:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:47:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:07,070 [231] [DEBUG] [app] Starting request: urn:request:0c18afb0-f0cd-4f7a-8351-3a2b93c7a7aa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:07,070 [236] [DEBUG] [app] Starting request: urn:request:55868a32-94e3-4171-ae2d-834a06f0862d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:07,070 [231] [DEBUG] [app] Ending request: urn:request:0c18afb0-f0cd-4f7a-8351-3a2b93c7a7aa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0c18afb0-f0cd-4f7a-8351-3a2b93c7a7aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:07,071 [236] [DEBUG] [app] Ending request: urn:request:55868a32-94e3-4171-ae2d-834a06f0862d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55868a32-94e3-4171-ae2d-834a06f0862d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:07,087 [236] [DEBUG] [app] Ending request: urn:request:de6165dc-b3d7-4d68-ad6c-bcc0f899a8c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de6165dc-b3d7-4d68-ad6c-bcc0f899a8c5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:07,087 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:47:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:47:07,095 [235] [DEBUG] [app] Ending request: urn:request:6d26bb9d-82cd-47d8-9750-5529a62058d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d26bb9d-82cd-47d8-9750-5529a62058d2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:07,095 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:07,095 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) repositoryactioncounter stdout | 2023-01-17 05:47:07,237 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:47:07,239 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:47:07,870 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:47:08,445 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:47:08,459 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:47:08,813 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:47:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:47:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:40.738568+00:00 (in 29.999522 seconds) buildlogsarchiver stdout | 2023-01-17 05:47:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:40 UTC)" (scheduled at 2023-01-17 05:47:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:47:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 47, 10, 739456), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:47:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:47:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:47:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:40 UTC)" executed successfully builder stdout | 2023-01-17 05:47:14,133 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:14,146 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:14,146 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:14,157 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:25.646354+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 05:47:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:25 UTC)" (scheduled at 2023-01-17 05:47:15.646354+00:00) notificationworker stdout | 2023-01-17 05:47:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 15, 647224), True, datetime.datetime(2023, 1, 17, 5, 47, 15, 647224), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:19.537472+00:00 (in 2.600051 seconds) securityworker stdout | 2023-01-17 05:47:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:46 UTC)" (scheduled at 2023-01-17 05:47:16.936988+00:00) securityworker stdout | 2023-01-17 05:47:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:47:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:47:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:04.936835+00:00 (in 45.590245 seconds) repositorygcworker stdout | 2023-01-17 05:47:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:19 UTC)" (scheduled at 2023-01-17 05:47:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:47:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:47:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:46.936988+00:00 (in 27.399047 seconds) securityworker stdout | 2023-01-17 05:47:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:49 UTC)" (scheduled at 2023-01-17 05:47:19.537472+00:00) securityworker stdout | 2023-01-17 05:47:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:47:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:47:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:47:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:47:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:47:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:47:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:47:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:47:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:51.348605+00:00 (in 29.999544 seconds) gcworker stdout | 2023-01-17 05:47:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:51 UTC)" (scheduled at 2023-01-17 05:47:21.348605+00:00) gcworker stdout | 2023-01-17 05:47:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724841349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:47:21,364 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:47:21,364 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:47:21,364 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:47:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:47:22,060 [233] [DEBUG] [app] Starting request: urn:request:05597c54-ee1f-41a9-9251-ef89c56232e2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:22,060 [231] [DEBUG] [app] Starting request: urn:request:6ac345ac-969d-44ff-ae26-964e23b6d7f1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:22,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:22,066 [230] [DEBUG] [app] Starting request: urn:request:65ff9218-1abf-497f-a4cd-290dfba32458 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:22,066 [242] [DEBUG] [app] Starting request: urn:request:81c569f3-77a9-4541-90e0-4b73d86cb1a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:22,066 [230] [DEBUG] [app] Ending request: urn:request:65ff9218-1abf-497f-a4cd-290dfba32458 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:65ff9218-1abf-497f-a4cd-290dfba32458', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:22,066 [242] [DEBUG] [app] Ending request: urn:request:81c569f3-77a9-4541-90e0-4b73d86cb1a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81c569f3-77a9-4541-90e0-4b73d86cb1a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:47:22,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:22,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:47:22,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:22,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:22,071 [235] [DEBUG] [app] Starting request: urn:request:81881ac3-b5ea-46d0-b429-bdb0094f2e09 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:22,071 [236] [DEBUG] [app] Starting request: urn:request:807c60f3-d923-4c21-925d-ab1e99d32878 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:22,071 [236] [DEBUG] [app] Ending request: urn:request:807c60f3-d923-4c21-925d-ab1e99d32878 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:807c60f3-d923-4c21-925d-ab1e99d32878', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:22,072 [235] [DEBUG] [app] Ending request: urn:request:81881ac3-b5ea-46d0-b429-bdb0094f2e09 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:81881ac3-b5ea-46d0-b429-bdb0094f2e09', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:22,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:22,088 [233] [DEBUG] [app] Ending request: urn:request:05597c54-ee1f-41a9-9251-ef89c56232e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05597c54-ee1f-41a9-9251-ef89c56232e2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:22,089 [231] [DEBUG] [app] Ending request: urn:request:6ac345ac-969d-44ff-ae26-964e23b6d7f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6ac345ac-969d-44ff-ae26-964e23b6d7f1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:22,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:47:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:47:24,167 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:24,179 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:24,179 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:24,189 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:35.646354+00:00 (in 9.999564 seconds) notificationworker stdout | 2023-01-17 05:47:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:35 UTC)" (scheduled at 2023-01-17 05:47:25.646354+00:00) notificationworker stdout | 2023-01-17 05:47:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 25, 647180), True, datetime.datetime(2023, 1, 17, 5, 47, 25, 647180), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:47:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:47:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:35.853396+00:00 (in 6.889164 seconds) exportactionlogsworker stdout | 2023-01-17 05:47:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:28 UTC)" (scheduled at 2023-01-17 05:47:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:47:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:47:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:47:29,710 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:47:29,712 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:47:30,622 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:47:31,206 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:47:31,208 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:47:31,281 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:47:31,284 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:47:31,286 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:47:31,288 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:47:31,291 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:47:31,363 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:47:31,755 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:47:34,200 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:34,212 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:34,212 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:34,223 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:47:34,810 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:47:34,897 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:47:35,274 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:47:35,277 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:47:35,279 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:47:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:45.646354+00:00 (in 9.999583 seconds) notificationworker stdout | 2023-01-17 05:47:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:45 UTC)" (scheduled at 2023-01-17 05:47:35.646354+00:00) notificationworker stdout | 2023-01-17 05:47:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 35, 647184), True, datetime.datetime(2023, 1, 17, 5, 47, 35, 647184), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:47:35,666 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:47:35,668 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:47:35,671 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:47:35,674 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:47:35,676 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:47:35,679 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:47:35,681 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:47:35,683 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:47:35,686 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:47:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:47:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:28.963662+00:00 (in 53.109762 seconds) exportactionlogsworker stdout | 2023-01-17 05:47:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:35 UTC)" (scheduled at 2023-01-17 05:47:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:47:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:47:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 35, 854268), True, datetime.datetime(2023, 1, 17, 5, 47, 35, 854268), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:47:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:47:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:47:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:47:36,039 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:47:36,104 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:47:36,312 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:47:36,872 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:47:37,059 [236] [DEBUG] [app] Starting request: urn:request:b85cf65c-11fb-473c-a3e7-a39aea842a43 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:37,060 [235] [DEBUG] [app] Starting request: urn:request:b8e29393-4c95-4598-b74e-a3a849108ca7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost manifestbackfillworker stdout | 2023-01-17 05:47:37,061 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 05:47:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:37,065 [242] [DEBUG] [app] Starting request: urn:request:34904823-8628-4014-aaea-04b1d821b9d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:37,065 [230] [DEBUG] [app] Starting request: urn:request:2bb016bc-5d79-44ec-8b26-8b68f93137f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:37,065 [242] [DEBUG] [app] Ending request: urn:request:34904823-8628-4014-aaea-04b1d821b9d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34904823-8628-4014-aaea-04b1d821b9d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:47:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:47:37,066 [230] [DEBUG] [app] Ending request: urn:request:2bb016bc-5d79-44ec-8b26-8b68f93137f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2bb016bc-5d79-44ec-8b26-8b68f93137f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:47:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:47:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:37,070 [233] [DEBUG] [app] Starting request: urn:request:bfcd4cc8-347c-4adb-87ba-1ff9cfc0f91f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:37,070 [233] [DEBUG] [app] Ending request: urn:request:bfcd4cc8-347c-4adb-87ba-1ff9cfc0f91f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bfcd4cc8-347c-4adb-87ba-1ff9cfc0f91f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:37,070 [236] [DEBUG] [app] Starting request: urn:request:78e81805-e2b6-40f9-bea4-0710871b72cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:37,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:47:37,071 [236] [DEBUG] [app] Ending request: urn:request:78e81805-e2b6-40f9-bea4-0710871b72cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78e81805-e2b6-40f9-bea4-0710871b72cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:37,088 [236] [DEBUG] [app] Ending request: urn:request:b85cf65c-11fb-473c-a3e7-a39aea842a43 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b85cf65c-11fb-473c-a3e7-a39aea842a43', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:37,088 [235] [DEBUG] [app] Ending request: urn:request:b8e29393-4c95-4598-b74e-a3a849108ca7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b8e29393-4c95-4598-b74e-a3a849108ca7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:47:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:47:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:47:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:37.537407+00:00 (in 0.280321 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:47:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:37 UTC)" (scheduled at 2023-01-17 05:47:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:47:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:47:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 37, 257493), True, datetime.datetime(2023, 1, 17, 5, 47, 37, 257493), 0, 'secscanv4/%', 50, 1, 0]) repositoryactioncounter stdout | 2023-01-17 05:47:37,260 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:47:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:47:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:47:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:47:37,272 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:47:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:47:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:37.256652+00:00 (in 59.718849 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:47:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:37 UTC)" (scheduled at 2023-01-17 05:47:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:47:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:47:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:47:37,903 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:47:38,453 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:47:38,479 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:47:38,845 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:47:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:47:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:10.738568+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2023-01-17 05:47:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:10 UTC)" (scheduled at 2023-01-17 05:47:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:47:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 47, 40, 739453), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:47:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:47:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:47:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:47:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:47:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:42.941536+00:00 (in 1.792757 seconds) namespacegcworker stdout | 2023-01-17 05:47:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:41 UTC)" (scheduled at 2023-01-17 05:47:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:47:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:47:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 41, 149211), True, datetime.datetime(2023, 1, 17, 5, 47, 41, 149211), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:47:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:47:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:47:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:47:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:47:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:41.148245+00:00 (in 58.206214 seconds) namespacegcworker stdout | 2023-01-17 05:47:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:42 UTC)" (scheduled at 2023-01-17 05:47:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:47:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:47:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:48:42 UTC)" executed successfully builder stdout | 2023-01-17 05:47:44,233 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:44,246 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:44,246 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:44,257 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:55.646354+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 05:47:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:55 UTC)" (scheduled at 2023-01-17 05:47:45.646354+00:00) notificationworker stdout | 2023-01-17 05:47:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 45, 647249), True, datetime.datetime(2023, 1, 17, 5, 47, 45, 647249), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:47:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:47:49.537472+00:00 (in 2.600017 seconds) securityworker stdout | 2023-01-17 05:47:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:16 UTC)" (scheduled at 2023-01-17 05:47:46.936988+00:00) securityworker stdout | 2023-01-17 05:47:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:46,955 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:46,958 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:46,958 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:47:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:47:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:16.936988+00:00 (in 27.399028 seconds) securityworker stdout | 2023-01-17 05:47:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:19 UTC)" (scheduled at 2023-01-17 05:47:49.537472+00:00) securityworker stdout | 2023-01-17 05:47:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:47:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:47:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:47:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:47:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:47:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:47:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:47:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:47:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:47:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:47:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:47:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:47:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:47:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:47:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:47:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:21.348605+00:00 (in 29.999514 seconds) gcworker stdout | 2023-01-17 05:47:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:21 UTC)" (scheduled at 2023-01-17 05:47:51.348605+00:00) gcworker stdout | 2023-01-17 05:47:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724871349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:47:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:47:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:47:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:47:52,060 [231] [DEBUG] [app] Starting request: urn:request:8b3d069c-6441-4786-a2e9-15bfd4c3cb8d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:52,060 [235] [DEBUG] [app] Starting request: urn:request:03c29692-a45c-4ed1-a3b5-c76484b10aa5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:47:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:52,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:47:52,065 [242] [DEBUG] [app] Starting request: urn:request:c12803d1-8fba-43d4-9110-435bde2474d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:52,066 [239] [DEBUG] [app] Starting request: urn:request:05bf0876-32ed-44db-b042-e0a0ebfb5aa3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:47:52,066 [242] [DEBUG] [app] Ending request: urn:request:c12803d1-8fba-43d4-9110-435bde2474d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c12803d1-8fba-43d4-9110-435bde2474d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:47:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:47:52,066 [239] [DEBUG] [app] Ending request: urn:request:05bf0876-32ed-44db-b042-e0a0ebfb5aa3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:05bf0876-32ed-44db-b042-e0a0ebfb5aa3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:47:52,067 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:47:52,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:47:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:47:52,071 [233] [DEBUG] [app] Starting request: urn:request:1a67cc40-5069-4cc3-8151-e1a3c79a8b65 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:52,071 [233] [DEBUG] [app] Ending request: urn:request:1a67cc40-5069-4cc3-8151-e1a3c79a8b65 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1a67cc40-5069-4cc3-8151-e1a3c79a8b65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:52,071 [236] [DEBUG] [app] Starting request: urn:request:c91578d1-08f5-4953-a60c-23ba5f7eb882 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:47:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:47:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:47:52,072 [236] [DEBUG] [app] Ending request: urn:request:c91578d1-08f5-4953-a60c-23ba5f7eb882 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c91578d1-08f5-4953-a60c-23ba5f7eb882', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:47:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:47:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:47:52,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:47:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:47:52,089 [235] [DEBUG] [app] Ending request: urn:request:03c29692-a45c-4ed1-a3b5-c76484b10aa5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:03c29692-a45c-4ed1-a3b5-c76484b10aa5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:52,089 [231] [DEBUG] [app] Ending request: urn:request:8b3d069c-6441-4786-a2e9-15bfd4c3cb8d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b3d069c-6441-4786-a2e9-15bfd4c3cb8d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:47:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:52,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:47:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:47:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:47:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 05:47:54,267 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:47:54,279 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:47:54,279 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:47:54,289 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:47:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:01.636362+00:00 (in 5.989523 seconds) notificationworker stdout | 2023-01-17 05:47:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:05 UTC)" (scheduled at 2023-01-17 05:47:55.646354+00:00) notificationworker stdout | 2023-01-17 05:47:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:47:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 47, 55, 647216), True, datetime.datetime(2023, 1, 17, 5, 47, 55, 647216), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:47:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:47:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:47:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:05 UTC)" executed successfully gcworker stdout | 2023-01-17 05:47:59,733 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 05:47:59,745 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} queuecleanupworker stdout | 2023-01-17 05:48:00,632 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:48:01,243 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:48:01,246 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:48:01,287 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:48:01,290 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:48:01,293 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:48:01,295 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:48:01,297 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:48:01,370 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:48:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:05.646354+00:00 (in 4.009532 seconds) notificationworker stdout | 2023-01-17 05:48:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:01 UTC)" (scheduled at 2023-01-17 05:48:01.636362+00:00) notificationworker stdout | 2023-01-17 05:48:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:48:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:01,791 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:48:04,300 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:04,312 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:04,312 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:04,323 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:48:04,845 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:48:04,931 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} repositorygcworker stdout | 2023-01-17 05:48:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:48:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:19.346120+00:00 (in 14.408867 seconds) repositorygcworker stdout | 2023-01-17 05:48:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:04 UTC)" (scheduled at 2023-01-17 05:48:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:48:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:48:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 4, 937659), True, datetime.datetime(2023, 1, 17, 5, 48, 4, 937659), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:48:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:48:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:48:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:48:05,282 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:48:05,285 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:48:05,287 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:48:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:15.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:48:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:15 UTC)" (scheduled at 2023-01-17 05:48:05.646354+00:00) notificationworker stdout | 2023-01-17 05:48:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 5, 647229), True, datetime.datetime(2023, 1, 17, 5, 48, 5, 647229), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:48:05,675 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:48:05,677 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:48:05,680 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:48:05,682 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:48:05,684 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:48:05,687 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:48:05,689 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:48:05,692 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:48:05,694 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:48:06,045 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:48:06,114 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:48:06,342 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:48:06,886 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:48:07,060 [231] [DEBUG] [app] Starting request: urn:request:15e4dd7e-26b7-4aa8-9678-93de54dd97c7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:07,060 [233] [DEBUG] [app] Starting request: urn:request:8185fbc2-c33e-4671-a380-c39ef00b5641 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:07,065 [242] [DEBUG] [app] Starting request: urn:request:1df671f7-d812-4f5c-9edf-afe4574d939a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:07,066 [241] [DEBUG] [app] Starting request: urn:request:efd3b345-131b-4c53-8885-1dc9417961ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:07,066 [242] [DEBUG] [app] Ending request: urn:request:1df671f7-d812-4f5c-9edf-afe4574d939a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1df671f7-d812-4f5c-9edf-afe4574d939a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:48:07,066 [241] [DEBUG] [app] Ending request: urn:request:efd3b345-131b-4c53-8885-1dc9417961ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:efd3b345-131b-4c53-8885-1dc9417961ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:48:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:48:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:07,071 [236] [DEBUG] [app] Starting request: urn:request:58c1d371-d7d5-46f1-b5e1-050a97b22e53 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:07,071 [235] [DEBUG] [app] Starting request: urn:request:58e5d093-789f-49a5-85d1-663450eed9c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:07,071 [236] [DEBUG] [app] Ending request: urn:request:58c1d371-d7d5-46f1-b5e1-050a97b22e53 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58c1d371-d7d5-46f1-b5e1-050a97b22e53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:07,071 [235] [DEBUG] [app] Ending request: urn:request:58e5d093-789f-49a5-85d1-663450eed9c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58e5d093-789f-49a5-85d1-663450eed9c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:48:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:07,088 [231] [DEBUG] [app] Ending request: urn:request:15e4dd7e-26b7-4aa8-9678-93de54dd97c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15e4dd7e-26b7-4aa8-9678-93de54dd97c7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:07,088 [233] [DEBUG] [app] Ending request: urn:request:8185fbc2-c33e-4671-a380-c39ef00b5641 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8185fbc2-c33e-4671-a380-c39ef00b5641', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:48:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) manifestbackfillworker stdout | 2023-01-17 05:48:07,096 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositoryactioncounter stdout | 2023-01-17 05:48:07,276 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:48:07,307 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:48:07,938 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:48:08,489 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:48:08,491 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:48:08,881 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:48:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:48:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:40.738568+00:00 (in 29.999501 seconds) buildlogsarchiver stdout | 2023-01-17 05:48:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:40 UTC)" (scheduled at 2023-01-17 05:48:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:48:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 48, 10, 739467), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:48:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:48:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:48:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:40 UTC)" executed successfully builder stdout | 2023-01-17 05:48:14,333 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:14,345 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:14,345 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:14,356 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:25.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 05:48:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:25 UTC)" (scheduled at 2023-01-17 05:48:15.646354+00:00) notificationworker stdout | 2023-01-17 05:48:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 15, 647214), True, datetime.datetime(2023, 1, 17, 5, 48, 15, 647214), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:19.537472+00:00 (in 2.600038 seconds) securityworker stdout | 2023-01-17 05:48:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:46 UTC)" (scheduled at 2023-01-17 05:48:16.936988+00:00) securityworker stdout | 2023-01-17 05:48:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:48:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:48:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:04.936835+00:00 (in 45.590279 seconds) repositorygcworker stdout | 2023-01-17 05:48:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:19 UTC)" (scheduled at 2023-01-17 05:48:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:48:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:48:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:46.936988+00:00 (in 27.399087 seconds) securityworker stdout | 2023-01-17 05:48:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:49 UTC)" (scheduled at 2023-01-17 05:48:19.537472+00:00) securityworker stdout | 2023-01-17 05:48:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:48:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:48:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:48:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:48:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:48:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:48:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:48:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:48:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:51.348605+00:00 (in 29.999575 seconds) gcworker stdout | 2023-01-17 05:48:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:51 UTC)" (scheduled at 2023-01-17 05:48:21.348605+00:00) gcworker stdout | 2023-01-17 05:48:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724901349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:48:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:48:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:48:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:48:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:48:22,060 [235] [DEBUG] [app] Starting request: urn:request:f041ef58-0821-40a5-9e8b-00a827e96193 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:22,060 [236] [DEBUG] [app] Starting request: urn:request:bc31df06-d0db-4b8b-ac5f-d0be00113b72 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:22,065 [230] [DEBUG] [app] Starting request: urn:request:b73e16e9-2bef-4fc7-a515-8fef14b925cf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:22,066 [243] [DEBUG] [app] Starting request: urn:request:1bfd317b-80ae-45b2-a106-9880808fcecb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:22,066 [230] [DEBUG] [app] Ending request: urn:request:b73e16e9-2bef-4fc7-a515-8fef14b925cf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b73e16e9-2bef-4fc7-a515-8fef14b925cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:22,066 [243] [DEBUG] [app] Ending request: urn:request:1bfd317b-80ae-45b2-a106-9880808fcecb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1bfd317b-80ae-45b2-a106-9880808fcecb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:48:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:48:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:48:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:22,070 [235] [DEBUG] [app] Starting request: urn:request:b7863e27-b405-4595-b3cb-3c48fdf2c93e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:22,071 [235] [DEBUG] [app] Ending request: urn:request:b7863e27-b405-4595-b3cb-3c48fdf2c93e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7863e27-b405-4595-b3cb-3c48fdf2c93e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:22,072 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:22,073 [231] [DEBUG] [app] Starting request: urn:request:05ca8839-9f81-4d08-b948-3f98ea394fce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:22,073 [231] [DEBUG] [app] Ending request: urn:request:05ca8839-9f81-4d08-b948-3f98ea394fce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05ca8839-9f81-4d08-b948-3f98ea394fce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:22,073 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:22,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:22,074 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:22,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:22,087 [236] [DEBUG] [app] Ending request: urn:request:bc31df06-d0db-4b8b-ac5f-d0be00113b72 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bc31df06-d0db-4b8b-ac5f-d0be00113b72', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:48:22,089 [235] [DEBUG] [app] Ending request: urn:request:f041ef58-0821-40a5-9e8b-00a827e96193 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f041ef58-0821-40a5-9e8b-00a827e96193', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 05:48:24,366 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:24,379 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:24,379 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:24,389 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:35.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 05:48:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:35 UTC)" (scheduled at 2023-01-17 05:48:25.646354+00:00) notificationworker stdout | 2023-01-17 05:48:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 25, 647249), True, datetime.datetime(2023, 1, 17, 5, 48, 25, 647249), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:48:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:48:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:35.853396+00:00 (in 6.889286 seconds) exportactionlogsworker stdout | 2023-01-17 05:48:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:28 UTC)" (scheduled at 2023-01-17 05:48:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:48:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:48:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:29,757 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:48:29,764 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:48:30,644 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:48:31,279 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:48:31,282 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:48:31,294 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:48:31,297 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:48:31,300 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:48:31,302 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:48:31,305 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:48:31,401 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:48:31,828 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:48:34,390 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:34,402 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:34,402 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:34,413 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:48:34,881 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:48:34,958 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:48:35,290 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:48:35,292 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:48:35,295 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:48:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:36.635997+00:00 (in 0.989157 seconds) notificationworker stdout | 2023-01-17 05:48:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:45 UTC)" (scheduled at 2023-01-17 05:48:35.646354+00:00) notificationworker stdout | 2023-01-17 05:48:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 35, 647267), True, datetime.datetime(2023, 1, 17, 5, 48, 35, 647267), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:48:35,683 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:48:35,685 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:48:35,688 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:48:35,690 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:48:35,693 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:48:35,695 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:48:35,699 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:48:35,702 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:48:35,704 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} exportactionlogsworker stdout | 2023-01-17 05:48:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:48:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:28.963662+00:00 (in 53.109800 seconds) exportactionlogsworker stdout | 2023-01-17 05:48:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:35 UTC)" (scheduled at 2023-01-17 05:48:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:48:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:48:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 35, 854267), True, datetime.datetime(2023, 1, 17, 5, 48, 35, 854267), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:48:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:48:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:48:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:48:36,080 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:48:36,149 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:48:36,377 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} notificationworker stdout | 2023-01-17 05:48:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:45.646354+00:00 (in 9.009882 seconds) notificationworker stdout | 2023-01-17 05:48:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:53:36 UTC)" (scheduled at 2023-01-17 05:48:36.635997+00:00) notificationworker stdout | 2023-01-17 05:48:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 48, 36, 636846), 'notification/%']) notificationworker stdout | 2023-01-17 05:48:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 48, 36, 636846), True, datetime.datetime(2023, 1, 17, 5, 48, 36, 636846), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:48:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 48, 36, 636846), True, datetime.datetime(2023, 1, 17, 5, 48, 36, 636846), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 48, 36, 636846), 'notification/%']) notificationworker stdout | 2023-01-17 05:48:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:53:36 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:48:36,899 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:48:37,059 [231] [DEBUG] [app] Starting request: urn:request:42845c4e-4086-458d-8e48-3d59be714709 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:37,059 [236] [DEBUG] [app] Starting request: urn:request:7bec0c8e-05eb-4b8a-8d42-5de7b1bf351b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:37,064 [241] [DEBUG] [app] Starting request: urn:request:c6e71b1a-5d77-4133-b739-938cadcc03a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,064 [243] [DEBUG] [app] Starting request: urn:request:6645a9fc-a213-4f09-a1ad-177aea0fc167 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,065 [241] [DEBUG] [app] Ending request: urn:request:c6e71b1a-5d77-4133-b739-938cadcc03a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c6e71b1a-5d77-4133-b739-938cadcc03a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,065 [243] [DEBUG] [app] Ending request: urn:request:6645a9fc-a213-4f09-a1ad-177aea0fc167 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6645a9fc-a213-4f09-a1ad-177aea0fc167', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:37,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:48:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:48:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:37,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:37,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:37,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:37,069 [233] [DEBUG] [app] Starting request: urn:request:6f7db529-bf5c-4a8b-b118-22d4e11df1e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:37,069 [235] [DEBUG] [app] Starting request: urn:request:ccf9c278-8823-47b9-97f5-3d765a3db237 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:37,069 [233] [DEBUG] [app] Ending request: urn:request:6f7db529-bf5c-4a8b-b118-22d4e11df1e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f7db529-bf5c-4a8b-b118-22d4e11df1e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:37,070 [235] [DEBUG] [app] Ending request: urn:request:ccf9c278-8823-47b9-97f5-3d765a3db237 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ccf9c278-8823-47b9-97f5-3d765a3db237', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:37,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:48:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:48:37,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:37,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:37,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:37,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:37,086 [236] [DEBUG] [app] Ending request: urn:request:7bec0c8e-05eb-4b8a-8d42-5de7b1bf351b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7bec0c8e-05eb-4b8a-8d42-5de7b1bf351b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:37,086 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:37,086 [231] [DEBUG] [app] Ending request: urn:request:42845c4e-4086-458d-8e48-3d59be714709 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42845c4e-4086-458d-8e48-3d59be714709', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:37,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:37,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.028) gunicorn-web stdout | 2023-01-17 05:48:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) manifestbackfillworker stdout | 2023-01-17 05:48:37,121 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:48:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:48:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:37.537407+00:00 (in 0.280330 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:48:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:37 UTC)" (scheduled at 2023-01-17 05:48:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:48:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:48:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 37, 257487), True, datetime.datetime(2023, 1, 17, 5, 48, 37, 257487), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:48:37,273 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:48:37,273 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:48:37,273 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:37 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:48:37,300 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:48:37,340 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:48:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:48:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:37.256652+00:00 (in 59.718801 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:48:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:37 UTC)" (scheduled at 2023-01-17 05:48:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:48:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:48:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:48:37,974 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:48:38,498 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:48:38,524 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:48:38,902 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:48:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:48:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:10.738568+00:00 (in 29.999560 seconds) buildlogsarchiver stdout | 2023-01-17 05:48:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:10 UTC)" (scheduled at 2023-01-17 05:48:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:48:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 48, 40, 739395), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:48:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:48:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:48:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:48:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:48:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:42.941536+00:00 (in 1.792782 seconds) namespacegcworker stdout | 2023-01-17 05:48:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:41 UTC)" (scheduled at 2023-01-17 05:48:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:48:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:48:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 41, 149188), True, datetime.datetime(2023, 1, 17, 5, 48, 41, 149188), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:48:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:48:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:48:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:48:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:48:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:41.148245+00:00 (in 58.206238 seconds) namespacegcworker stdout | 2023-01-17 05:48:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:42 UTC)" (scheduled at 2023-01-17 05:48:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:48:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:48:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:49:42 UTC)" executed successfully builder stdout | 2023-01-17 05:48:44,423 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:44,436 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:44,436 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:44,447 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:55.646354+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 05:48:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:55 UTC)" (scheduled at 2023-01-17 05:48:45.646354+00:00) notificationworker stdout | 2023-01-17 05:48:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 45, 647187), True, datetime.datetime(2023, 1, 17, 5, 48, 45, 647187), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:48:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:48:49.537472+00:00 (in 2.600046 seconds) securityworker stdout | 2023-01-17 05:48:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:16 UTC)" (scheduled at 2023-01-17 05:48:46.936988+00:00) securityworker stdout | 2023-01-17 05:48:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:48:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:48:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:16.936988+00:00 (in 27.399031 seconds) securityworker stdout | 2023-01-17 05:48:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:19 UTC)" (scheduled at 2023-01-17 05:48:49.537472+00:00) securityworker stdout | 2023-01-17 05:48:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:48:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:48:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:48:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:48:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:48:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:48:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:48:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:48:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:48:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:48:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:48:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:48:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:48:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:48:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:48:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:21.348605+00:00 (in 29.999530 seconds) gcworker stdout | 2023-01-17 05:48:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:21 UTC)" (scheduled at 2023-01-17 05:48:51.348605+00:00) gcworker stdout | 2023-01-17 05:48:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724931349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:48:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:48:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:48:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:48:52,060 [231] [DEBUG] [app] Starting request: urn:request:377f3cf5-8db1-4fec-81e5-2c4f5aff7909 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:52,060 [236] [DEBUG] [app] Starting request: urn:request:905f081c-b0b3-4a3a-b530-1e9b8aa00e56 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:48:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:48:52,065 [243] [DEBUG] [app] Starting request: urn:request:b610220b-1373-4920-a2bd-c5d6c3c17688 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:52,065 [230] [DEBUG] [app] Starting request: urn:request:a9a0ba2f-3b4e-4ae8-b2f7-c8d60b718a8b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:48:52,066 [243] [DEBUG] [app] Ending request: urn:request:b610220b-1373-4920-a2bd-c5d6c3c17688 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b610220b-1373-4920-a2bd-c5d6c3c17688', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:52,066 [230] [DEBUG] [app] Ending request: urn:request:a9a0ba2f-3b4e-4ae8-b2f7-c8d60b718a8b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9a0ba2f-3b4e-4ae8-b2f7-c8d60b718a8b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:48:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:48:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:48:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:48:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:48:52,071 [233] [DEBUG] [app] Starting request: urn:request:425949ce-3593-4017-8f3b-e4cccb2d504c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:52,071 [235] [DEBUG] [app] Starting request: urn:request:3a5caaa3-88be-439d-9b10-6fe8802985a1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:48:52,071 [233] [DEBUG] [app] Ending request: urn:request:425949ce-3593-4017-8f3b-e4cccb2d504c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:425949ce-3593-4017-8f3b-e4cccb2d504c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:52,071 [235] [DEBUG] [app] Ending request: urn:request:3a5caaa3-88be-439d-9b10-6fe8802985a1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3a5caaa3-88be-439d-9b10-6fe8802985a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:48:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:48:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:48:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:48:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:48:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:48:52,089 [231] [DEBUG] [app] Ending request: urn:request:377f3cf5-8db1-4fec-81e5-2c4f5aff7909 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:377f3cf5-8db1-4fec-81e5-2c4f5aff7909', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:52,089 [236] [DEBUG] [app] Ending request: urn:request:905f081c-b0b3-4a3a-b530-1e9b8aa00e56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:905f081c-b0b3-4a3a-b530-1e9b8aa00e56', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:48:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:48:52,089 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:48:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:48:52,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:48:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:48:54,457 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:48:54,470 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:48:54,470 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:48:54,481 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:48:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:01.636362+00:00 (in 5.989557 seconds) notificationworker stdout | 2023-01-17 05:48:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:05 UTC)" (scheduled at 2023-01-17 05:48:55.646354+00:00) notificationworker stdout | 2023-01-17 05:48:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:48:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 48, 55, 647199), True, datetime.datetime(2023, 1, 17, 5, 48, 55, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:48:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:48:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:48:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:48:59,783 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:48:59,799 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:49:00,673 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 05:49:01,287 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:49:01,300 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:49:01,305 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:49:01,307 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} builder stdout | 2023-01-17 05:49:01,310 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} gunicorn-web stdout | 2023-01-17 05:49:01,313 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:49:01,315 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} namespacegcworker stdout | 2023-01-17 05:49:01,436 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:49:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:05.646354+00:00 (in 4.009566 seconds) notificationworker stdout | 2023-01-17 05:49:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:01 UTC)" (scheduled at 2023-01-17 05:49:01.636362+00:00) notificationworker stdout | 2023-01-17 05:49:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:49:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:01,839 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:49:04,491 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:04,504 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:04,504 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:04,515 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:49:04,917 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} repositorygcworker stdout | 2023-01-17 05:49:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:49:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:19.346120+00:00 (in 14.408873 seconds) repositorygcworker stdout | 2023-01-17 05:49:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:04 UTC)" (scheduled at 2023-01-17 05:49:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:49:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:49:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 4, 937700), True, datetime.datetime(2023, 1, 17, 5, 49, 4, 937700), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:49:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:49:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:49:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:04 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:49:04,986 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:49:05,297 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:49:05,299 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:49:05,302 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:49:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:15.646354+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 05:49:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:15 UTC)" (scheduled at 2023-01-17 05:49:05.646354+00:00) notificationworker stdout | 2023-01-17 05:49:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 5, 647171), True, datetime.datetime(2023, 1, 17, 5, 49, 5, 647171), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:49:05,694 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:49:05,697 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:49:05,699 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:49:05,701 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:49:05,704 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:49:05,706 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:49:05,709 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:49:05,711 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:49:05,714 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} storagereplication stdout | 2023-01-17 05:49:06,087 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:49:06,182 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:49:06,390 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:49:06,934 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:49:07,060 [231] [DEBUG] [app] Starting request: urn:request:06d84abf-7f82-474b-a328-f39330e12655 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:07,061 [235] [DEBUG] [app] Starting request: urn:request:e7b0ea78-cce4-4d2b-b890-102f65c59a56 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:07,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:07,065 [230] [DEBUG] [app] Starting request: urn:request:e89aceb9-ae8a-4234-a709-11052df96740 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:07,066 [230] [DEBUG] [app] Ending request: urn:request:e89aceb9-ae8a-4234-a709-11052df96740 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e89aceb9-ae8a-4234-a709-11052df96740', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:49:07,066 [243] [DEBUG] [app] Starting request: urn:request:71c15c73-4951-4833-9a03-7e599be61440 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:49:07,066 [243] [DEBUG] [app] Ending request: urn:request:71c15c73-4951-4833-9a03-7e599be61440 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71c15c73-4951-4833-9a03-7e599be61440', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:07,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:49:07,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:07,070 [236] [DEBUG] [app] Starting request: urn:request:b97b0c91-6c76-4a43-88eb-d77b43376b4b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:07,071 [236] [DEBUG] [app] Ending request: urn:request:b97b0c91-6c76-4a43-88eb-d77b43376b4b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b97b0c91-6c76-4a43-88eb-d77b43376b4b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:07,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:49:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:07,072 [233] [DEBUG] [app] Starting request: urn:request:c489b782-358c-483a-baf2-cef81ad0b396 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:07,072 [233] [DEBUG] [app] Ending request: urn:request:c489b782-358c-483a-baf2-cef81ad0b396 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c489b782-358c-483a-baf2-cef81ad0b396', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:07,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:07,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:07,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:07,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:07,088 [231] [DEBUG] [app] Ending request: urn:request:06d84abf-7f82-474b-a328-f39330e12655 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:06d84abf-7f82-474b-a328-f39330e12655', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:49:07,089 [235] [DEBUG] [app] Ending request: urn:request:e7b0ea78-cce4-4d2b-b890-102f65c59a56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e7b0ea78-cce4-4d2b-b890-102f65c59a56', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:07,089 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:49:07,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" manifestbackfillworker stdout | 2023-01-17 05:49:07,127 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositoryactioncounter stdout | 2023-01-17 05:49:07,312 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:49:07,347 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:49:08,010 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:49:08,525 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:49:08,545 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:49:08,916 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:49:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:49:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:40.738568+00:00 (in 29.999544 seconds) buildlogsarchiver stdout | 2023-01-17 05:49:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:40 UTC)" (scheduled at 2023-01-17 05:49:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:49:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 49, 10, 739446), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:49:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:49:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:49:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:40 UTC)" executed successfully builder stdout | 2023-01-17 05:49:14,525 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:14,538 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:14,538 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:14,549 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:25.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 05:49:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:25 UTC)" (scheduled at 2023-01-17 05:49:15.646354+00:00) notificationworker stdout | 2023-01-17 05:49:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 15, 647241), True, datetime.datetime(2023, 1, 17, 5, 49, 15, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:19.537472+00:00 (in 2.599999 seconds) securityworker stdout | 2023-01-17 05:49:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:46 UTC)" (scheduled at 2023-01-17 05:49:16.936988+00:00) securityworker stdout | 2023-01-17 05:49:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:49:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:49:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:04.936835+00:00 (in 45.590226 seconds) repositorygcworker stdout | 2023-01-17 05:49:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:19 UTC)" (scheduled at 2023-01-17 05:49:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:49:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:49:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:46.936988+00:00 (in 27.399024 seconds) securityworker stdout | 2023-01-17 05:49:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:49 UTC)" (scheduled at 2023-01-17 05:49:19.537472+00:00) securityworker stdout | 2023-01-17 05:49:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:49:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:49:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:49:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:49:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:49:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:49:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:49:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:49:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:51.348605+00:00 (in 29.999516 seconds) gcworker stdout | 2023-01-17 05:49:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:51 UTC)" (scheduled at 2023-01-17 05:49:21.348605+00:00) gcworker stdout | 2023-01-17 05:49:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724961349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:49:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:49:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:49:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:49:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:49:22,059 [231] [DEBUG] [app] Starting request: urn:request:7a704a5d-911e-46ce-888f-f6dcdd461cf5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:22,059 [236] [DEBUG] [app] Starting request: urn:request:82ecc8b8-d125-4dbc-8251-ba1fabbb4bad (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:22,064 [243] [DEBUG] [app] Starting request: urn:request:43b98bff-0cdd-4b9f-95a2-b0f14dc37a64 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:22,065 [239] [DEBUG] [app] Starting request: urn:request:d65c54da-2a22-4e7d-8ed8-f03459fdad07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:22,065 [243] [DEBUG] [app] Ending request: urn:request:43b98bff-0cdd-4b9f-95a2-b0f14dc37a64 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:43b98bff-0cdd-4b9f-95a2-b0f14dc37a64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:49:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:49:22,065 [239] [DEBUG] [app] Ending request: urn:request:d65c54da-2a22-4e7d-8ed8-f03459fdad07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d65c54da-2a22-4e7d-8ed8-f03459fdad07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:49:22,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:49:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:22,069 [233] [DEBUG] [app] Starting request: urn:request:92607de1-ceff-40d9-85a4-1239a4a40d14 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:22,070 [233] [DEBUG] [app] Ending request: urn:request:92607de1-ceff-40d9-85a4-1239a4a40d14 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92607de1-ceff-40d9-85a4-1239a4a40d14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:22,070 [235] [DEBUG] [app] Starting request: urn:request:27a08f4c-ae81-434b-bf20-513c1489ffb9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:49:22,071 [235] [DEBUG] [app] Ending request: urn:request:27a08f4c-ae81-434b-bf20-513c1489ffb9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27a08f4c-ae81-434b-bf20-513c1489ffb9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:49:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:22,087 [231] [DEBUG] [app] Ending request: urn:request:7a704a5d-911e-46ce-888f-f6dcdd461cf5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a704a5d-911e-46ce-888f-f6dcdd461cf5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:22,087 [236] [DEBUG] [app] Ending request: urn:request:82ecc8b8-d125-4dbc-8251-ba1fabbb4bad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:82ecc8b8-d125-4dbc-8251-ba1fabbb4bad', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:49:22,087 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:49:22,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) builder stdout | 2023-01-17 05:49:24,555 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:24,568 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:24,568 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:24,578 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:35.646354+00:00 (in 9.999509 seconds) notificationworker stdout | 2023-01-17 05:49:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:35 UTC)" (scheduled at 2023-01-17 05:49:25.646354+00:00) notificationworker stdout | 2023-01-17 05:49:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 25, 647297), True, datetime.datetime(2023, 1, 17, 5, 49, 25, 647297), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:49:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:49:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:35.853396+00:00 (in 6.889244 seconds) exportactionlogsworker stdout | 2023-01-17 05:49:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:28 UTC)" (scheduled at 2023-01-17 05:49:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:49:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:49:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:29,810 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:49:29,832 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:49:30,701 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:49:31,308 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:49:31,311 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:49:31,314 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:49:31,319 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} logrotateworker stdout | 2023-01-17 05:49:31,321 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} gunicorn-web stdout | 2023-01-17 05:49:31,323 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:49:31,340 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:49:31,453 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:49:31,846 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:49:34,588 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:34,601 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:34,601 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:34,612 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:49:34,953 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:49:35,018 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:49:35,304 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:49:35,307 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:49:35,309 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:49:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:45.646354+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 05:49:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:45 UTC)" (scheduled at 2023-01-17 05:49:35.646354+00:00) notificationworker stdout | 2023-01-17 05:49:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 35, 647292), True, datetime.datetime(2023, 1, 17, 5, 49, 35, 647292), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:49:35,705 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:49:35,708 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:49:35,712 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:49:35,714 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:49:35,716 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:49:35,719 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:49:35,721 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:49:35,724 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 05:49:35,726 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} exportactionlogsworker stdout | 2023-01-17 05:49:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:49:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:28.963662+00:00 (in 53.109773 seconds) exportactionlogsworker stdout | 2023-01-17 05:49:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:35 UTC)" (scheduled at 2023-01-17 05:49:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:49:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:49:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 35, 854362), True, datetime.datetime(2023, 1, 17, 5, 49, 35, 854362), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:49:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:49:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:49:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:49:36,094 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:49:36,213 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:49:36,425 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:49:36,970 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:49:37,060 [235] [DEBUG] [app] Starting request: urn:request:21c2703d-b004-4a28-9936-0b9ea4b1b922 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:37,060 [233] [DEBUG] [app] Starting request: urn:request:3fdd03aa-7bfb-4858-9b7d-aee4b327f372 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:37,065 [230] [DEBUG] [app] Starting request: urn:request:6485ff44-1b7f-41a6-9feb-8d791d2a498d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,065 [240] [DEBUG] [app] Starting request: urn:request:7d9cf8ba-7f62-455e-ba10-b55d488cfc62 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,065 [230] [DEBUG] [app] Ending request: urn:request:6485ff44-1b7f-41a6-9feb-8d791d2a498d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6485ff44-1b7f-41a6-9feb-8d791d2a498d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,066 [240] [DEBUG] [app] Ending request: urn:request:7d9cf8ba-7f62-455e-ba10-b55d488cfc62 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7d9cf8ba-7f62-455e-ba10-b55d488cfc62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:49:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:49:37,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:37,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:37,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:37,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:37,071 [231] [DEBUG] [app] Starting request: urn:request:64c173d1-60d4-42df-b70a-f4648c2d2459 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:37,071 [231] [DEBUG] [app] Ending request: urn:request:64c173d1-60d4-42df-b70a-f4648c2d2459 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64c173d1-60d4-42df-b70a-f4648c2d2459', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:37,071 [236] [DEBUG] [app] Starting request: urn:request:90a01026-9c03-46e3-883f-88da51366364 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:37,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:37,072 [236] [DEBUG] [app] Ending request: urn:request:90a01026-9c03-46e3-883f-88da51366364 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:90a01026-9c03-46e3-883f-88da51366364', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:49:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:37,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:37,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:37,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:37,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:37,089 [235] [DEBUG] [app] Ending request: urn:request:21c2703d-b004-4a28-9936-0b9ea4b1b922 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21c2703d-b004-4a28-9936-0b9ea4b1b922', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:37,089 [233] [DEBUG] [app] Ending request: urn:request:3fdd03aa-7bfb-4858-9b7d-aee4b327f372 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3fdd03aa-7bfb-4858-9b7d-aee4b327f372', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:37,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:37,089 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.030) gunicorn-web stdout | 2023-01-17 05:49:37,090 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:49:37,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" manifestbackfillworker stdout | 2023-01-17 05:49:37,139 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:49:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:49:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:37.537407+00:00 (in 0.280335 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:49:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:37 UTC)" (scheduled at 2023-01-17 05:49:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:49:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:49:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 37, 257449), True, datetime.datetime(2023, 1, 17, 5, 49, 37, 257449), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:49:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:49:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:49:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:37 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:49:37,344 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:49:37,381 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:49:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:49:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:37.256652+00:00 (in 59.718898 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:49:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:37 UTC)" (scheduled at 2023-01-17 05:49:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:49:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:49:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:49:38,038 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:49:38,543 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:49:38,580 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:49:38,934 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:49:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:49:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:10.738568+00:00 (in 29.999554 seconds) buildlogsarchiver stdout | 2023-01-17 05:49:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:10 UTC)" (scheduled at 2023-01-17 05:49:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:49:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 49, 40, 739428), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:49:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:49:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:49:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:49:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:49:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:42.941536+00:00 (in 1.792832 seconds) namespacegcworker stdout | 2023-01-17 05:49:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:41 UTC)" (scheduled at 2023-01-17 05:49:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:49:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:49:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 41, 149133), True, datetime.datetime(2023, 1, 17, 5, 49, 41, 149133), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:49:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:49:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:49:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:49:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:49:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:41.148245+00:00 (in 58.206250 seconds) namespacegcworker stdout | 2023-01-17 05:49:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:42 UTC)" (scheduled at 2023-01-17 05:49:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:49:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:49:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:50:42 UTC)" executed successfully builder stdout | 2023-01-17 05:49:44,618 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:44,630 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:44,631 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:44,641 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:55.646354+00:00 (in 9.999539 seconds) notificationworker stdout | 2023-01-17 05:49:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:55 UTC)" (scheduled at 2023-01-17 05:49:45.646354+00:00) notificationworker stdout | 2023-01-17 05:49:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 45, 647226), True, datetime.datetime(2023, 1, 17, 5, 49, 45, 647226), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:49:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:49:49.537472+00:00 (in 2.599991 seconds) securityworker stdout | 2023-01-17 05:49:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:16 UTC)" (scheduled at 2023-01-17 05:49:46.936988+00:00) securityworker stdout | 2023-01-17 05:49:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:49:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:49:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:16.936988+00:00 (in 27.399019 seconds) securityworker stdout | 2023-01-17 05:49:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:19 UTC)" (scheduled at 2023-01-17 05:49:49.537472+00:00) securityworker stdout | 2023-01-17 05:49:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:49:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:49:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:49:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:49:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:49:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:49:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:49:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:49:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:49:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:49:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:49:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:49:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:49:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:49:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:49:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:21.348605+00:00 (in 29.999527 seconds) gcworker stdout | 2023-01-17 05:49:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:21 UTC)" (scheduled at 2023-01-17 05:49:51.348605+00:00) gcworker stdout | 2023-01-17 05:49:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672724991349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:49:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:49:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:49:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:49:52,060 [235] [DEBUG] [app] Starting request: urn:request:7edbd801-257d-4355-af4a-38b0d30fe720 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:52,060 [233] [DEBUG] [app] Starting request: urn:request:a77b3746-69e9-4d23-b5f1-59bf3ff75bb5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:49:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:49:52,065 [243] [DEBUG] [app] Starting request: urn:request:3193aebd-b332-4d2b-9931-d2a2796c1dd4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:52,065 [240] [DEBUG] [app] Starting request: urn:request:3ad29cc1-1118-4fa9-857c-a965b9352cf9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:49:52,066 [240] [DEBUG] [app] Ending request: urn:request:3ad29cc1-1118-4fa9-857c-a965b9352cf9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ad29cc1-1118-4fa9-857c-a965b9352cf9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:49:52,066 [243] [DEBUG] [app] Ending request: urn:request:3193aebd-b332-4d2b-9931-d2a2796c1dd4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3193aebd-b332-4d2b-9931-d2a2796c1dd4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:49:52,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:49:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:52,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:49:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:52,071 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:49:52,071 [231] [DEBUG] [app] Starting request: urn:request:bf6903d0-3a71-469b-ab31-4d7e200e1915 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:52,072 [231] [DEBUG] [app] Ending request: urn:request:bf6903d0-3a71-469b-ab31-4d7e200e1915 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf6903d0-3a71-469b-ab31-4d7e200e1915', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:52,072 [236] [DEBUG] [app] Starting request: urn:request:0085cab3-28f7-4bf1-94c0-79273545cbf1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:49:52,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:49:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:49:52,072 [236] [DEBUG] [app] Ending request: urn:request:0085cab3-28f7-4bf1-94c0-79273545cbf1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0085cab3-28f7-4bf1-94c0-79273545cbf1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:49:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:49:52,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:52,073 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:49:52,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:49:52,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:52,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:49:52,089 [235] [DEBUG] [app] Ending request: urn:request:7edbd801-257d-4355-af4a-38b0d30fe720 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7edbd801-257d-4355-af4a-38b0d30fe720', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:52,089 [233] [DEBUG] [app] Ending request: urn:request:a77b3746-69e9-4d23-b5f1-59bf3ff75bb5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a77b3746-69e9-4d23-b5f1-59bf3ff75bb5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:49:52,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:49:52,089 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 05:49:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:49:52,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:49:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:49:54,651 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:49:54,664 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:49:54,664 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:49:54,675 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:49:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:01.636362+00:00 (in 5.989523 seconds) notificationworker stdout | 2023-01-17 05:49:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:05 UTC)" (scheduled at 2023-01-17 05:49:55.646354+00:00) notificationworker stdout | 2023-01-17 05:49:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:49:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 49, 55, 647258), True, datetime.datetime(2023, 1, 17, 5, 49, 55, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:49:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:49:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:49:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:49:59,845 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:49:59,866 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:50:00,707 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:50:01,315 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:50:01,318 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:50:01,321 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:50:01,325 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:50:01,329 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:50:01,350 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:50:01,355 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:50:01,489 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:50:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:05.646354+00:00 (in 4.009505 seconds) notificationworker stdout | 2023-01-17 05:50:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:01 UTC)" (scheduled at 2023-01-17 05:50:01.636362+00:00) notificationworker stdout | 2023-01-17 05:50:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:50:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:01,868 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:50:04,685 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:04,706 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:04,706 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:04,716 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:50:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:50:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:19.346120+00:00 (in 14.408858 seconds) repositorygcworker stdout | 2023-01-17 05:50:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:04 UTC)" (scheduled at 2023-01-17 05:50:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:50:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:50:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 4, 937691), True, datetime.datetime(2023, 1, 17, 5, 50, 4, 937691), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:50:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:50:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:50:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:04 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:50:04,985 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:50:05,053 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:50:05,312 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:50:05,314 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:50:05,317 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:50:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:15.646354+00:00 (in 9.999497 seconds) notificationworker stdout | 2023-01-17 05:50:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:15 UTC)" (scheduled at 2023-01-17 05:50:05.646354+00:00) notificationworker stdout | 2023-01-17 05:50:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 5, 647255), True, datetime.datetime(2023, 1, 17, 5, 50, 5, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:50:05,716 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:50:05,719 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:50:05,721 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:50:05,724 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:50:05,726 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:50:05,730 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:50:05,732 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:50:05,735 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:50:05,738 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:50:06,103 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:50:06,226 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:50:06,453 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:50:06,990 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:50:07,060 [231] [DEBUG] [app] Starting request: urn:request:ab8d61ff-c3cd-45d0-b2c5-bafcf1c4cd9f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:07,060 [236] [DEBUG] [app] Starting request: urn:request:c4d07ee1-ba1a-49cf-b7db-8310d7e35aa6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:07,065 [230] [DEBUG] [app] Starting request: urn:request:3fa8274e-fdba-4a0d-af92-ed90976adb6c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:07,065 [243] [DEBUG] [app] Starting request: urn:request:bb3878c5-9396-427b-93ff-d47e179dbe70 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:07,066 [230] [DEBUG] [app] Ending request: urn:request:3fa8274e-fdba-4a0d-af92-ed90976adb6c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3fa8274e-fdba-4a0d-af92-ed90976adb6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:07,066 [243] [DEBUG] [app] Ending request: urn:request:bb3878c5-9396-427b-93ff-d47e179dbe70 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bb3878c5-9396-427b-93ff-d47e179dbe70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:50:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:50:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:07,070 [233] [DEBUG] [app] Starting request: urn:request:9b032508-8157-440e-9cad-0c9f39b3db8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:07,071 [235] [DEBUG] [app] Starting request: urn:request:9b627824-c501-46e3-95f1-64ce2b77d0d8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:07,071 [233] [DEBUG] [app] Ending request: urn:request:9b032508-8157-440e-9cad-0c9f39b3db8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b032508-8157-440e-9cad-0c9f39b3db8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:07,071 [235] [DEBUG] [app] Ending request: urn:request:9b627824-c501-46e3-95f1-64ce2b77d0d8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b627824-c501-46e3-95f1-64ce2b77d0d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:07,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:07,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:07,090 [236] [DEBUG] [app] Ending request: urn:request:c4d07ee1-ba1a-49cf-b7db-8310d7e35aa6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4d07ee1-ba1a-49cf-b7db-8310d7e35aa6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:07,090 [231] [DEBUG] [app] Ending request: urn:request:ab8d61ff-c3cd-45d0-b2c5-bafcf1c4cd9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab8d61ff-c3cd-45d0-b2c5-bafcf1c4cd9f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:07,090 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:07,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:07,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:50:07,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) manifestbackfillworker stdout | 2023-01-17 05:50:07,175 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositoryactioncounter stdout | 2023-01-17 05:50:07,378 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:50:07,399 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:50:08,064 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:50:08,578 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:50:08,585 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:50:08,970 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:50:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:50:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:40.738568+00:00 (in 29.999538 seconds) buildlogsarchiver stdout | 2023-01-17 05:50:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:40 UTC)" (scheduled at 2023-01-17 05:50:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:50:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 50, 10, 739436), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:50:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:50:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:50:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:40 UTC)" executed successfully builder stdout | 2023-01-17 05:50:14,725 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:14,738 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:14,738 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:14,748 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:25.646354+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:50:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:25 UTC)" (scheduled at 2023-01-17 05:50:15.646354+00:00) notificationworker stdout | 2023-01-17 05:50:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 15, 647203), True, datetime.datetime(2023, 1, 17, 5, 50, 15, 647203), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:19.537472+00:00 (in 2.600053 seconds) securityworker stdout | 2023-01-17 05:50:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:46 UTC)" (scheduled at 2023-01-17 05:50:16.936988+00:00) securityworker stdout | 2023-01-17 05:50:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:50:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:50:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:04.936835+00:00 (in 45.590277 seconds) repositorygcworker stdout | 2023-01-17 05:50:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:19 UTC)" (scheduled at 2023-01-17 05:50:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:50:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:50:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:46.936988+00:00 (in 27.399041 seconds) securityworker stdout | 2023-01-17 05:50:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:49 UTC)" (scheduled at 2023-01-17 05:50:19.537472+00:00) securityworker stdout | 2023-01-17 05:50:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:50:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:50:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:50:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:50:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:50:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:50:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:50:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:50:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:51.348605+00:00 (in 29.999478 seconds) gcworker stdout | 2023-01-17 05:50:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:51 UTC)" (scheduled at 2023-01-17 05:50:21.348605+00:00) gcworker stdout | 2023-01-17 05:50:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725021349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:50:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:50:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:50:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:50:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:50:22,060 [236] [DEBUG] [app] Starting request: urn:request:bcc0ce07-fe63-4c87-a02f-b6a60209df70 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:22,060 [235] [DEBUG] [app] Starting request: urn:request:8356c256-8c21-4ec0-8cca-62c53176da73 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:22,065 [230] [DEBUG] [app] Starting request: urn:request:678d2407-c367-401a-945f-9241f5836600 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:22,065 [241] [DEBUG] [app] Starting request: urn:request:b1ba5702-8da0-4c0e-8496-95c1ea8b836d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:22,065 [230] [DEBUG] [app] Ending request: urn:request:678d2407-c367-401a-945f-9241f5836600 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:678d2407-c367-401a-945f-9241f5836600', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:50:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:50:22,066 [241] [DEBUG] [app] Ending request: urn:request:b1ba5702-8da0-4c0e-8496-95c1ea8b836d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1ba5702-8da0-4c0e-8496-95c1ea8b836d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:50:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:50:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:22,070 [233] [DEBUG] [app] Starting request: urn:request:c7ff4c11-7b3b-4075-9da9-0d33c2573483 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:22,071 [235] [DEBUG] [app] Starting request: urn:request:09dbcc90-7eb0-4349-b1b7-95e13eb6ed32 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:22,071 [233] [DEBUG] [app] Ending request: urn:request:c7ff4c11-7b3b-4075-9da9-0d33c2573483 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c7ff4c11-7b3b-4075-9da9-0d33c2573483', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:22,071 [235] [DEBUG] [app] Ending request: urn:request:09dbcc90-7eb0-4349-b1b7-95e13eb6ed32 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:09dbcc90-7eb0-4349-b1b7-95e13eb6ed32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:50:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:22,088 [235] [DEBUG] [app] Ending request: urn:request:8356c256-8c21-4ec0-8cca-62c53176da73 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8356c256-8c21-4ec0-8cca-62c53176da73', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:50:22,094 [236] [DEBUG] [app] Ending request: urn:request:bcc0ce07-fe63-4c87-a02f-b6a60209df70 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bcc0ce07-fe63-4c87-a02f-b6a60209df70', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:22,094 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:22,095 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) builder stdout | 2023-01-17 05:50:24,759 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:24,773 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:24,774 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:24,784 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:35.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 05:50:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:35 UTC)" (scheduled at 2023-01-17 05:50:25.646354+00:00) notificationworker stdout | 2023-01-17 05:50:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 25, 647245), True, datetime.datetime(2023, 1, 17, 5, 50, 25, 647245), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:50:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:50:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:35.853396+00:00 (in 6.889275 seconds) exportactionlogsworker stdout | 2023-01-17 05:50:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:28 UTC)" (scheduled at 2023-01-17 05:50:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:50:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:50:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:29,871 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:50:29,901 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:50:30,743 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:50:31,323 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:50:31,326 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:50:31,328 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:50:31,332 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:50:31,335 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:50:31,360 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:50:31,365 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:50:31,496 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:50:31,904 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:50:34,794 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:34,807 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:34,807 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:34,817 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:50:35,021 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:50:35,088 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:50:35,319 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:50:35,322 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:50:35,324 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:50:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:45.646354+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 05:50:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:45 UTC)" (scheduled at 2023-01-17 05:50:35.646354+00:00) notificationworker stdout | 2023-01-17 05:50:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 35, 647198), True, datetime.datetime(2023, 1, 17, 5, 50, 35, 647198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:50:35,727 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:50:35,730 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:50:35,732 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:50:35,735 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:50:35,737 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:50:35,740 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:50:35,743 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:50:35,745 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:50:35,748 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:50:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:50:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:28.963662+00:00 (in 53.109775 seconds) exportactionlogsworker stdout | 2023-01-17 05:50:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:35 UTC)" (scheduled at 2023-01-17 05:50:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:50:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:50:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 35, 854272), True, datetime.datetime(2023, 1, 17, 5, 50, 35, 854272), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:50:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:50:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:50:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:50:36,138 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:50:36,252 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:50:36,488 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:50:37,026 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:50:37,059 [236] [DEBUG] [app] Starting request: urn:request:560796da-82b4-4bcc-8f46-ef06d1a68022 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:37,059 [235] [DEBUG] [app] Starting request: urn:request:2f26750d-539c-4f2a-b760-759ffbbc3f51 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:37,064 [230] [DEBUG] [app] Starting request: urn:request:377921ea-070f-4ea8-8f0b-1c5f407110e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:37,065 [230] [DEBUG] [app] Ending request: urn:request:377921ea-070f-4ea8-8f0b-1c5f407110e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:377921ea-070f-4ea8-8f0b-1c5f407110e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:37,065 [243] [DEBUG] [app] Starting request: urn:request:0294d716-048c-4914-9520-1be514659a3f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:50:37,065 [243] [DEBUG] [app] Ending request: urn:request:0294d716-048c-4914-9520-1be514659a3f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0294d716-048c-4914-9520-1be514659a3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:50:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:50:37,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:37,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:37,069 [231] [DEBUG] [app] Starting request: urn:request:cef1bb9f-0f23-4617-bd71-be675444729d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:37,070 [231] [DEBUG] [app] Ending request: urn:request:cef1bb9f-0f23-4617-bd71-be675444729d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cef1bb9f-0f23-4617-bd71-be675444729d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:37,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:37,070 [233] [DEBUG] [app] Starting request: urn:request:db9025d4-2300-43ba-9591-e8cde97a4f1f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:37,071 [233] [DEBUG] [app] Ending request: urn:request:db9025d4-2300-43ba-9591-e8cde97a4f1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db9025d4-2300-43ba-9591-e8cde97a4f1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:37,087 [236] [DEBUG] [app] Ending request: urn:request:560796da-82b4-4bcc-8f46-ef06d1a68022 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:560796da-82b4-4bcc-8f46-ef06d1a68022', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:50:37,088 [235] [DEBUG] [app] Ending request: urn:request:2f26750d-539c-4f2a-b760-759ffbbc3f51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f26750d-539c-4f2a-b760-759ffbbc3f51', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) manifestbackfillworker stdout | 2023-01-17 05:50:37,205 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:50:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:50:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:37.537407+00:00 (in 0.280356 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:50:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:37 UTC)" (scheduled at 2023-01-17 05:50:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:50:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:50:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 37, 257422), True, datetime.datetime(2023, 1, 17, 5, 50, 37, 257422), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:50:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:50:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:50:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:37 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:50:37,413 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:50:37,425 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:50:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:50:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:23.537048+00:00 (in 45.999303 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:50:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:37 UTC)" (scheduled at 2023-01-17 05:50:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:50:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:50:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:50:38,087 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:50:38,614 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:50:38,619 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:50:39,006 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:50:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:50:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:10.738568+00:00 (in 29.999505 seconds) buildlogsarchiver stdout | 2023-01-17 05:50:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:10 UTC)" (scheduled at 2023-01-17 05:50:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:50:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 50, 40, 739469), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:50:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:50:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:50:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:50:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:50:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:42.941536+00:00 (in 1.792840 seconds) namespacegcworker stdout | 2023-01-17 05:50:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:41 UTC)" (scheduled at 2023-01-17 05:50:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:50:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:50:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 41, 149092), True, datetime.datetime(2023, 1, 17, 5, 50, 41, 149092), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:50:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:50:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:50:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:50:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:50:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:41.148245+00:00 (in 58.206264 seconds) namespacegcworker stdout | 2023-01-17 05:50:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:42 UTC)" (scheduled at 2023-01-17 05:50:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:50:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:50:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:51:42 UTC)" executed successfully builder stdout | 2023-01-17 05:50:44,827 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:44,840 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:44,840 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:44,851 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:55.646354+00:00 (in 9.999569 seconds) notificationworker stdout | 2023-01-17 05:50:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:55 UTC)" (scheduled at 2023-01-17 05:50:45.646354+00:00) notificationworker stdout | 2023-01-17 05:50:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 45, 647190), True, datetime.datetime(2023, 1, 17, 5, 50, 45, 647190), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:50:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:50:49.537472+00:00 (in 2.599999 seconds) securityworker stdout | 2023-01-17 05:50:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:16 UTC)" (scheduled at 2023-01-17 05:50:46.936988+00:00) securityworker stdout | 2023-01-17 05:50:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:50:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:50:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:16.936988+00:00 (in 27.399014 seconds) securityworker stdout | 2023-01-17 05:50:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:19 UTC)" (scheduled at 2023-01-17 05:50:49.537472+00:00) securityworker stdout | 2023-01-17 05:50:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:50:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:50:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:50:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:50:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:50:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:50:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:50:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:50:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:50:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:50:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:50:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:50:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:50:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:50:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:50:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:21.348605+00:00 (in 29.999484 seconds) gcworker stdout | 2023-01-17 05:50:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:21 UTC)" (scheduled at 2023-01-17 05:50:51.348605+00:00) gcworker stdout | 2023-01-17 05:50:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725051349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:50:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:50:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:50:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:50:52,060 [231] [DEBUG] [app] Starting request: urn:request:b35823c4-a5fd-4cac-841d-f505dbac7a5f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:52,060 [235] [DEBUG] [app] Starting request: urn:request:09279887-7cc0-4d72-ab8d-1a99fefabd7f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:50:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:52,064 [230] [DEBUG] [app] Starting request: urn:request:8a3acfd2-a2be-48bc-84e7-615508d7209c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:50:52,065 [230] [DEBUG] [app] Ending request: urn:request:8a3acfd2-a2be-48bc-84e7-615508d7209c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a3acfd2-a2be-48bc-84e7-615508d7209c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:50:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:50:52,065 [243] [DEBUG] [app] Starting request: urn:request:be280c72-cf4e-48aa-880d-7cbb4fd9c397 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:50:52,066 [243] [DEBUG] [app] Ending request: urn:request:be280c72-cf4e-48aa-880d-7cbb4fd9c397 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be280c72-cf4e-48aa-880d-7cbb4fd9c397', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:50:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:50:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:50:52,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:52,069 [236] [DEBUG] [app] Starting request: urn:request:6311a927-95ad-4d37-9ab7-33c40111256a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:52,070 [236] [DEBUG] [app] Ending request: urn:request:6311a927-95ad-4d37-9ab7-33c40111256a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6311a927-95ad-4d37-9ab7-33c40111256a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:52,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:50:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:50:52,071 [236] [DEBUG] [app] Starting request: urn:request:fa98a356-cbea-47e3-b48a-9f584275dcc3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:50:52,071 [236] [DEBUG] [app] Ending request: urn:request:fa98a356-cbea-47e3-b48a-9f584275dcc3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa98a356-cbea-47e3-b48a-9f584275dcc3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:50:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:50:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:50:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:50:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:50:52,087 [231] [DEBUG] [app] Ending request: urn:request:b35823c4-a5fd-4cac-841d-f505dbac7a5f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b35823c4-a5fd-4cac-841d-f505dbac7a5f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:50:52,088 [235] [DEBUG] [app] Ending request: urn:request:09279887-7cc0-4d72-ab8d-1a99fefabd7f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09279887-7cc0-4d72-ab8d-1a99fefabd7f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:50:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:50:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:50:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:50:54,851 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:50:54,864 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:50:54,864 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:50:54,874 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:50:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:01.636362+00:00 (in 5.989545 seconds) notificationworker stdout | 2023-01-17 05:50:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:05 UTC)" (scheduled at 2023-01-17 05:50:55.646354+00:00) notificationworker stdout | 2023-01-17 05:50:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:50:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 50, 55, 647243), True, datetime.datetime(2023, 1, 17, 5, 50, 55, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:50:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:50:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:50:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:50:59,885 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:50:59,933 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:51:00,775 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:51:01,331 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:51:01,333 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:51:01,336 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:51:01,338 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:51:01,341 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 05:51:01,385 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 05:51:01,388 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 05:51:01,502 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:51:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:05.646354+00:00 (in 4.009527 seconds) notificationworker stdout | 2023-01-17 05:51:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:01 UTC)" (scheduled at 2023-01-17 05:51:01.636362+00:00) notificationworker stdout | 2023-01-17 05:51:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:51:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:51:01,940 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:51:04,884 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:04,897 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:04,897 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:04,908 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:51:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:51:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:17.345601+00:00 (in 12.408347 seconds) repositorygcworker stdout | 2023-01-17 05:51:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:04 UTC)" (scheduled at 2023-01-17 05:51:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:51:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:51:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 4, 937737), True, datetime.datetime(2023, 1, 17, 5, 51, 4, 937737), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:51:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:51:04,948 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:51:04,948 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:04 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 05:51:05,056 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:51:05,094 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:51:05,326 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:51:05,328 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:51:05,330 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:51:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:15.646354+00:00 (in 9.999453 seconds) notificationworker stdout | 2023-01-17 05:51:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:15 UTC)" (scheduled at 2023-01-17 05:51:05.646354+00:00) notificationworker stdout | 2023-01-17 05:51:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 5, 647363), True, datetime.datetime(2023, 1, 17, 5, 51, 5, 647363), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:51:05,737 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:51:05,739 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:51:05,742 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:51:05,745 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:51:05,747 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:51:05,749 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:51:05,752 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:51:05,754 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:51:05,756 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:51:06,171 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:51:06,286 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:51:06,524 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 05:51:07,055 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:51:07,060 [236] [DEBUG] [app] Starting request: urn:request:26e0dc4c-39f2-46ac-b86d-a69833764f90 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:07,060 [233] [DEBUG] [app] Starting request: urn:request:b20c9518-787e-4525-89ee-9f54f7537ec9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:07,065 [243] [DEBUG] [app] Starting request: urn:request:e761210c-f922-4802-923e-bdf8ce28d022 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:07,065 [242] [DEBUG] [app] Starting request: urn:request:e03cd4a1-9942-4882-95f2-e86cc31dc545 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:07,065 [243] [DEBUG] [app] Ending request: urn:request:e761210c-f922-4802-923e-bdf8ce28d022 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e761210c-f922-4802-923e-bdf8ce28d022', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:07,066 [242] [DEBUG] [app] Ending request: urn:request:e03cd4a1-9942-4882-95f2-e86cc31dc545 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e03cd4a1-9942-4882-95f2-e86cc31dc545', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:51:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:51:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:07,070 [231] [DEBUG] [app] Starting request: urn:request:e1cdda20-17bb-4de1-b098-87480d7f46e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:07,071 [233] [DEBUG] [app] Starting request: urn:request:847df987-9085-45fc-9c50-dc57d89a30a0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:07,071 [231] [DEBUG] [app] Ending request: urn:request:e1cdda20-17bb-4de1-b098-87480d7f46e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e1cdda20-17bb-4de1-b098-87480d7f46e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:07,071 [233] [DEBUG] [app] Ending request: urn:request:847df987-9085-45fc-9c50-dc57d89a30a0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:847df987-9085-45fc-9c50-dc57d89a30a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:07,088 [233] [DEBUG] [app] Ending request: urn:request:b20c9518-787e-4525-89ee-9f54f7537ec9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b20c9518-787e-4525-89ee-9f54f7537ec9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:07,088 [236] [DEBUG] [app] Ending request: urn:request:26e0dc4c-39f2-46ac-b86d-a69833764f90 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26e0dc4c-39f2-46ac-b86d-a69833764f90', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:51:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" manifestbackfillworker stdout | 2023-01-17 05:51:07,226 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositoryactioncounter stdout | 2023-01-17 05:51:07,438 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:51:07,441 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:51:08,122 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:51:08,650 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:51:08,654 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:51:09,026 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:51:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:51:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:40.738568+00:00 (in 29.999514 seconds) buildlogsarchiver stdout | 2023-01-17 05:51:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:40 UTC)" (scheduled at 2023-01-17 05:51:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:51:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 51, 10, 739446), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:51:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:51:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:51:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:40 UTC)" executed successfully builder stdout | 2023-01-17 05:51:14,914 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:14,927 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:14,927 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:14,937 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:25.646354+00:00 (in 9.999485 seconds) notificationworker stdout | 2023-01-17 05:51:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:25 UTC)" (scheduled at 2023-01-17 05:51:15.646354+00:00) notificationworker stdout | 2023-01-17 05:51:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 15, 647336), True, datetime.datetime(2023, 1, 17, 5, 51, 15, 647336), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:51:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:19.537472+00:00 (in 2.600039 seconds) securityworker stdout | 2023-01-17 05:51:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:46 UTC)" (scheduled at 2023-01-17 05:51:16.936988+00:00) securityworker stdout | 2023-01-17 05:51:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:16,957 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:16,960 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:16,960 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:51:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:51:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:19.346120+00:00 (in 2.000008 seconds) repositorygcworker stdout | 2023-01-17 05:51:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:17 UTC)" (scheduled at 2023-01-17 05:51:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:51:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 51, 17, 346524), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:51:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 17, 346524), True, datetime.datetime(2023, 1, 17, 5, 51, 17, 346524), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:51:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 17, 346524), True, datetime.datetime(2023, 1, 17, 5, 51, 17, 346524), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 51, 17, 346524), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:51:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:51:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:51:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:51:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:04.936835+00:00 (in 45.590238 seconds) repositorygcworker stdout | 2023-01-17 05:51:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:19 UTC)" (scheduled at 2023-01-17 05:51:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:51:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:51:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:51:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:46.936988+00:00 (in 27.399058 seconds) securityworker stdout | 2023-01-17 05:51:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:49 UTC)" (scheduled at 2023-01-17 05:51:19.537472+00:00) securityworker stdout | 2023-01-17 05:51:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:51:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:51:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:19,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:51:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:51:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:51:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:51:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:51:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:51:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:51.348605+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 05:51:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:51 UTC)" (scheduled at 2023-01-17 05:51:21.348605+00:00) gcworker stdout | 2023-01-17 05:51:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725081349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:51:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:51:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:51:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:51:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:51:22,060 [231] [DEBUG] [app] Starting request: urn:request:136363fc-1acd-4690-9879-45e7663e74bb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:22,060 [236] [DEBUG] [app] Starting request: urn:request:4a57ee7c-af4e-4a05-bec4-5b0973a39a6b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:22,065 [230] [DEBUG] [app] Starting request: urn:request:9e8710b2-f0c3-4082-a8c7-cf2ed939649d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:22,066 [243] [DEBUG] [app] Starting request: urn:request:5bdeca71-c165-4695-b373-051a7e398b3f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:22,066 [230] [DEBUG] [app] Ending request: urn:request:9e8710b2-f0c3-4082-a8c7-cf2ed939649d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e8710b2-f0c3-4082-a8c7-cf2ed939649d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:51:22,066 [243] [DEBUG] [app] Ending request: urn:request:5bdeca71-c165-4695-b373-051a7e398b3f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5bdeca71-c165-4695-b373-051a7e398b3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:51:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:22,071 [233] [DEBUG] [app] Starting request: urn:request:8b1b8bb2-940f-4fed-9c35-3ece7ad423c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:22,071 [236] [DEBUG] [app] Starting request: urn:request:67cc3048-9b22-4bf7-b8bf-8e4a3231d634 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:22,071 [233] [DEBUG] [app] Ending request: urn:request:8b1b8bb2-940f-4fed-9c35-3ece7ad423c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b1b8bb2-940f-4fed-9c35-3ece7ad423c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:22,071 [236] [DEBUG] [app] Ending request: urn:request:67cc3048-9b22-4bf7-b8bf-8e4a3231d634 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:67cc3048-9b22-4bf7-b8bf-8e4a3231d634', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:22,088 [236] [DEBUG] [app] Ending request: urn:request:4a57ee7c-af4e-4a05-bec4-5b0973a39a6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4a57ee7c-af4e-4a05-bec4-5b0973a39a6b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:22,089 [231] [DEBUG] [app] Ending request: urn:request:136363fc-1acd-4690-9879-45e7663e74bb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:136363fc-1acd-4690-9879-45e7663e74bb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:51:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 05:51:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:51:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:37.256652+00:00 (in 13.719162 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:51:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:23 UTC)" (scheduled at 2023-01-17 05:51:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:51:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 51, 23, 537877), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:51:23,559 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 23, 537877), True, datetime.datetime(2023, 1, 17, 5, 51, 23, 537877), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:51:23,563 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 23, 537877), True, datetime.datetime(2023, 1, 17, 5, 51, 23, 537877), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 51, 23, 537877), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:51:23,567 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:51:23,567 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:23 UTC)" executed successfully builder stdout | 2023-01-17 05:51:24,947 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:24,960 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:24,960 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:24,971 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:35.646354+00:00 (in 9.999493 seconds) notificationworker stdout | 2023-01-17 05:51:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:35 UTC)" (scheduled at 2023-01-17 05:51:25.646354+00:00) notificationworker stdout | 2023-01-17 05:51:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 25, 647273), True, datetime.datetime(2023, 1, 17, 5, 51, 25, 647273), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:51:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:51:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:35.853396+00:00 (in 6.889300 seconds) exportactionlogsworker stdout | 2023-01-17 05:51:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:28 UTC)" (scheduled at 2023-01-17 05:51:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:51:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:51:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:29,912 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:51:29,947 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:51:30,785 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:51:31,338 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:51:31,342 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:51:31,345 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:51:31,347 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:51:31,349 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:51:31,397 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:51:31,419 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:51:31,507 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:51:31,952 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 05:51:34,981 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:34,993 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:34,993 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:35,004 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:51:35,092 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:51:35,121 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:51:35,332 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:51:35,335 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:51:35,337 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:51:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:45.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:51:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:45 UTC)" (scheduled at 2023-01-17 05:51:35.646354+00:00) notificationworker stdout | 2023-01-17 05:51:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 35, 647241), True, datetime.datetime(2023, 1, 17, 5, 51, 35, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:51:35,747 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:51:35,750 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:51:35,753 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:51:35,755 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:51:35,757 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:51:35,760 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:51:35,762 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:51:35,764 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:51:35,767 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:51:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:51:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:54.963413+00:00 (in 19.109569 seconds) exportactionlogsworker stdout | 2023-01-17 05:51:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:35 UTC)" (scheduled at 2023-01-17 05:51:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:51:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:51:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 35, 854231), True, datetime.datetime(2023, 1, 17, 5, 51, 35, 854231), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:51:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:51:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:51:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:51:36,206 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:51:36,318 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:51:36,559 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:51:37,060 [231] [DEBUG] [app] Starting request: urn:request:d6fc9ee3-819f-4c04-b582-2c2329d820e5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:37,060 [233] [DEBUG] [app] Starting request: urn:request:08219646-f100-45a0-9989-fcd0997ca4c4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:37,065 [230] [DEBUG] [app] Starting request: urn:request:8b410ab1-184b-4cf6-810e-b9aa030516d3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:37,065 [243] [DEBUG] [app] Starting request: urn:request:da52e6cc-c1bd-4cac-b339-042550018af6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:37,065 [230] [DEBUG] [app] Ending request: urn:request:8b410ab1-184b-4cf6-810e-b9aa030516d3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b410ab1-184b-4cf6-810e-b9aa030516d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:51:37,066 [243] [DEBUG] [app] Ending request: urn:request:da52e6cc-c1bd-4cac-b339-042550018af6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da52e6cc-c1bd-4cac-b339-042550018af6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:51:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:51:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:37,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:37,070 [236] [DEBUG] [app] Starting request: urn:request:46e181c9-9622-4df9-9774-c236f32b009f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:37,071 [235] [DEBUG] [app] Starting request: urn:request:c4be1b9b-e6a7-410e-8d4b-43d7a605465c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:37,071 [236] [DEBUG] [app] Ending request: urn:request:46e181c9-9622-4df9-9774-c236f32b009f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:46e181c9-9622-4df9-9774-c236f32b009f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:37,071 [235] [DEBUG] [app] Ending request: urn:request:c4be1b9b-e6a7-410e-8d4b-43d7a605465c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4be1b9b-e6a7-410e-8d4b-43d7a605465c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:51:37,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:51:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:37,082 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:37,088 [233] [DEBUG] [app] Ending request: urn:request:08219646-f100-45a0-9989-fcd0997ca4c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08219646-f100-45a0-9989-fcd0997ca4c4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:37,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:37,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) chunkcleanupworker stdout | 2023-01-17 05:51:37,090 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} gunicorn-web stdout | 2023-01-17 05:51:37,091 [231] [DEBUG] [app] Ending request: urn:request:d6fc9ee3-819f-4c04-b582-2c2329d820e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d6fc9ee3-819f-4c04-b582-2c2329d820e5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:37,091 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:37,091 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) manifestbackfillworker stdout | 2023-01-17 05:51:37,237 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:51:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:51:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:37.537407+00:00 (in 0.280306 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:51:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:37 UTC)" (scheduled at 2023-01-17 05:51:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:51:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:51:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 37, 257497), True, datetime.datetime(2023, 1, 17, 5, 51, 37, 257497), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:51:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:51:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:51:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:37 UTC)" executed successfully repositoryactioncounter stdout | 2023-01-17 05:51:37,464 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:51:37,475 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:51:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:51:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:37.256652+00:00 (in 59.718866 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:51:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:37 UTC)" (scheduled at 2023-01-17 05:51:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:51:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:51:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:51:38,148 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:51:38,683 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:51:38,686 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:51:39,049 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:51:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:51:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:10.738568+00:00 (in 29.999523 seconds) buildlogsarchiver stdout | 2023-01-17 05:51:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:10 UTC)" (scheduled at 2023-01-17 05:51:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:51:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 51, 40, 739424), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:51:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:51:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:51:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:51:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:51:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:42.941536+00:00 (in 1.792830 seconds) namespacegcworker stdout | 2023-01-17 05:51:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:41 UTC)" (scheduled at 2023-01-17 05:51:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:51:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:51:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 41, 149113), True, datetime.datetime(2023, 1, 17, 5, 51, 41, 149113), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:51:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:51:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:51:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:51:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:51:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:06.941185+00:00 (in 23.999201 seconds) namespacegcworker stdout | 2023-01-17 05:51:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:42 UTC)" (scheduled at 2023-01-17 05:51:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:51:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:51:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:52:42 UTC)" executed successfully builder stdout | 2023-01-17 05:51:45,014 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:45,026 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:45,026 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:45,037 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:55.646354+00:00 (in 9.999492 seconds) notificationworker stdout | 2023-01-17 05:51:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:55 UTC)" (scheduled at 2023-01-17 05:51:45.646354+00:00) notificationworker stdout | 2023-01-17 05:51:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 45, 647351), True, datetime.datetime(2023, 1, 17, 5, 51, 45, 647351), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:51:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:51:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:51:49.537472+00:00 (in 2.600037 seconds) securityworker stdout | 2023-01-17 05:51:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:16 UTC)" (scheduled at 2023-01-17 05:51:46.936988+00:00) securityworker stdout | 2023-01-17 05:51:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:51:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:51:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:16.936988+00:00 (in 27.399054 seconds) securityworker stdout | 2023-01-17 05:51:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:19 UTC)" (scheduled at 2023-01-17 05:51:49.537472+00:00) securityworker stdout | 2023-01-17 05:51:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:51:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:51:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:51:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:51:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:51:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:51:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:51:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:51:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:51:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:51:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:51:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:51:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:51:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:51:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:51:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:21.348605+00:00 (in 29.999544 seconds) gcworker stdout | 2023-01-17 05:51:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:21 UTC)" (scheduled at 2023-01-17 05:51:51.348605+00:00) gcworker stdout | 2023-01-17 05:51:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725111349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:51:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:51:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:51:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:51:52,060 [231] [DEBUG] [app] Starting request: urn:request:aee46628-0313-4db8-bf64-483389859999 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:52,060 [236] [DEBUG] [app] Starting request: urn:request:090e6aa9-ebd3-4fbd-abee-02c154757a01 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:51:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:51:52,065 [243] [DEBUG] [app] Starting request: urn:request:83b60dff-5079-4664-a1e7-368426b98041 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:52,065 [239] [DEBUG] [app] Starting request: urn:request:09e3663f-945c-4cc3-92cc-7d25fd835f60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:51:52,065 [243] [DEBUG] [app] Ending request: urn:request:83b60dff-5079-4664-a1e7-368426b98041 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:83b60dff-5079-4664-a1e7-368426b98041', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:51:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:51:52,066 [239] [DEBUG] [app] Ending request: urn:request:09e3663f-945c-4cc3-92cc-7d25fd835f60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09e3663f-945c-4cc3-92cc-7d25fd835f60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:51:52,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:51:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:51:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:52,070 [231] [DEBUG] [app] Starting request: urn:request:b4c4e3d4-9685-4997-8867-f84adf3bed3e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:52,070 [231] [DEBUG] [app] Ending request: urn:request:b4c4e3d4-9685-4997-8867-f84adf3bed3e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4c4e3d4-9685-4997-8867-f84adf3bed3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:52,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:51:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:52,072 [235] [DEBUG] [app] Starting request: urn:request:a520c057-6ae5-40a6-85ff-549ea407904e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:51:52,072 [235] [DEBUG] [app] Ending request: urn:request:a520c057-6ae5-40a6-85ff-549ea407904e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a520c057-6ae5-40a6-85ff-549ea407904e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:51:52,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:51:52,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:51:52,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:51:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:51:52,087 [236] [DEBUG] [app] Ending request: urn:request:090e6aa9-ebd3-4fbd-abee-02c154757a01 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:090e6aa9-ebd3-4fbd-abee-02c154757a01', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:51:52,088 [231] [DEBUG] [app] Ending request: urn:request:aee46628-0313-4db8-bf64-483389859999 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aee46628-0313-4db8-bf64-483389859999', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:51:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:51:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:51:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) exportactionlogsworker stdout | 2023-01-17 05:51:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:51:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:28.963662+00:00 (in 33.999774 seconds) exportactionlogsworker stdout | 2023-01-17 05:51:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:54 UTC)" (scheduled at 2023-01-17 05:51:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:51:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 51, 54, 964378), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:51:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 54, 964378), True, datetime.datetime(2023, 1, 17, 5, 51, 54, 964378), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:51:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 51, 54, 964378), True, datetime.datetime(2023, 1, 17, 5, 51, 54, 964378), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 51, 54, 964378), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:51:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:51:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:56:54 UTC)" executed successfully builder stdout | 2023-01-17 05:51:55,044 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:51:55,062 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:51:55,063 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:51:55,073 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:51:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:01.636362+00:00 (in 5.989530 seconds) notificationworker stdout | 2023-01-17 05:51:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:05 UTC)" (scheduled at 2023-01-17 05:51:55.646354+00:00) notificationworker stdout | 2023-01-17 05:51:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:51:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 51, 55, 647272), True, datetime.datetime(2023, 1, 17, 5, 51, 55, 647272), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:51:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:51:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:51:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:51:59,934 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:51:59,967 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:52:00,816 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:52:01,345 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:52:01,348 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:52:01,351 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:52:01,353 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:52:01,356 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:52:01,422 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:52:01,440 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:52:01,515 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:52:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:05.646354+00:00 (in 4.009522 seconds) notificationworker stdout | 2023-01-17 05:52:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:01 UTC)" (scheduled at 2023-01-17 05:52:01.636362+00:00) notificationworker stdout | 2023-01-17 05:52:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:52:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:52:01,959 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:52:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:52:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:19.346120+00:00 (in 14.408839 seconds) repositorygcworker stdout | 2023-01-17 05:52:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:04 UTC)" (scheduled at 2023-01-17 05:52:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:52:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:52:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 4, 937719), True, datetime.datetime(2023, 1, 17, 5, 52, 4, 937719), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:52:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:52:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:52:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:04 UTC)" executed successfully builder stdout | 2023-01-17 05:52:05,083 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:05,095 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:05,095 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:05,105 [60] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 05:52:05,122 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:52:05,126 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:52:05,340 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:52:05,342 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:52:05,345 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:52:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:15.646354+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 05:52:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:15 UTC)" (scheduled at 2023-01-17 05:52:05.646354+00:00) notificationworker stdout | 2023-01-17 05:52:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 5, 647216), True, datetime.datetime(2023, 1, 17, 5, 52, 5, 647216), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:52:05,756 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:52:05,758 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:52:05,761 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:52:05,763 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:52:05,766 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:52:05,768 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:52:05,771 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:52:05,773 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:52:05,776 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:52:06,214 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:52:06,336 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:52:06,565 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} namespacegcworker stdout | 2023-01-17 05:52:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:52:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:41.148245+00:00 (in 34.206583 seconds) namespacegcworker stdout | 2023-01-17 05:52:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:57:06 UTC)" (scheduled at 2023-01-17 05:52:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:52:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 52, 6, 941989), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:52:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 52, 6, 941989), True, datetime.datetime(2023, 1, 17, 5, 52, 6, 941989), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:52:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 52, 6, 941989), True, datetime.datetime(2023, 1, 17, 5, 52, 6, 941989), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 52, 6, 941989), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:52:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:52:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:57:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:52:07,060 [231] [DEBUG] [app] Starting request: urn:request:b8b23e8a-95c5-4453-9c08-37c382dc67bd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:07,060 [233] [DEBUG] [app] Starting request: urn:request:16770d76-0678-4cef-a692-74cce9884e25 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:07,065 [230] [DEBUG] [app] Starting request: urn:request:33045e41-0766-474b-b676-30310875e49c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:07,065 [240] [DEBUG] [app] Starting request: urn:request:c4594f35-dcf9-4815-b526-f3439102379f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:07,065 [230] [DEBUG] [app] Ending request: urn:request:33045e41-0766-474b-b676-30310875e49c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:33045e41-0766-474b-b676-30310875e49c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:07,066 [240] [DEBUG] [app] Ending request: urn:request:c4594f35-dcf9-4815-b526-f3439102379f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c4594f35-dcf9-4815-b526-f3439102379f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:52:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:07,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:07,071 [235] [DEBUG] [app] Starting request: urn:request:1296f33a-b2d7-49e6-8785-d23400311000 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:07,071 [231] [DEBUG] [app] Starting request: urn:request:1e1e6739-a37b-4fd5-aef5-f831205932f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:07,071 [235] [DEBUG] [app] Ending request: urn:request:1296f33a-b2d7-49e6-8785-d23400311000 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1296f33a-b2d7-49e6-8785-d23400311000', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:07,071 [231] [DEBUG] [app] Ending request: urn:request:1e1e6739-a37b-4fd5-aef5-f831205932f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e1e6739-a37b-4fd5-aef5-f831205932f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:07,087 [233] [DEBUG] [app] Ending request: urn:request:16770d76-0678-4cef-a692-74cce9884e25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16770d76-0678-4cef-a692-74cce9884e25', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:07,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:07,087 [231] [DEBUG] [app] Ending request: urn:request:b8b23e8a-95c5-4453-9c08-37c382dc67bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b8b23e8a-95c5-4453-9c08-37c382dc67bd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:52:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 05:52:07,126 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:52:07,253 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositoryactioncounter stdout | 2023-01-17 05:52:07,489 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:52:07,492 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} globalpromstats stdout | 2023-01-17 05:52:08,181 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} exportactionlogsworker stdout | 2023-01-17 05:52:08,720 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} teamsyncworker stdout | 2023-01-17 05:52:08,722 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} blobuploadcleanupworker stdout | 2023-01-17 05:52:09,084 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:52:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:52:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:40.738568+00:00 (in 29.999538 seconds) buildlogsarchiver stdout | 2023-01-17 05:52:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:40 UTC)" (scheduled at 2023-01-17 05:52:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:52:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 52, 10, 739458), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:52:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:52:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:52:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:40 UTC)" executed successfully builder stdout | 2023-01-17 05:52:15,108 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:15,121 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:15,121 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:15,131 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:25.646354+00:00 (in 9.999558 seconds) notificationworker stdout | 2023-01-17 05:52:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:25 UTC)" (scheduled at 2023-01-17 05:52:15.646354+00:00) notificationworker stdout | 2023-01-17 05:52:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 15, 647160), True, datetime.datetime(2023, 1, 17, 5, 52, 15, 647160), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:52:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:19.537472+00:00 (in 2.599969 seconds) securityworker stdout | 2023-01-17 05:52:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:46 UTC)" (scheduled at 2023-01-17 05:52:16.936988+00:00) securityworker stdout | 2023-01-17 05:52:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:52:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:52:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:04.936835+00:00 (in 45.590274 seconds) repositorygcworker stdout | 2023-01-17 05:52:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:19 UTC)" (scheduled at 2023-01-17 05:52:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:52:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:52:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:52:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:46.936988+00:00 (in 27.399029 seconds) securityworker stdout | 2023-01-17 05:52:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:49 UTC)" (scheduled at 2023-01-17 05:52:19.537472+00:00) securityworker stdout | 2023-01-17 05:52:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:52:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:52:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:52:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:52:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:52:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:52:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:52:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:52:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:51.348605+00:00 (in 29.999563 seconds) gcworker stdout | 2023-01-17 05:52:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:51 UTC)" (scheduled at 2023-01-17 05:52:21.348605+00:00) gcworker stdout | 2023-01-17 05:52:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725141349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:52:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:52:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:52:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:52:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:52:22,060 [236] [DEBUG] [app] Starting request: urn:request:93f70443-850b-4161-be82-fc5832217d17 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:22,060 [231] [DEBUG] [app] Starting request: urn:request:3189b6c3-3c52-4d81-b12f-3b3ec83e8ea0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:22,064 [230] [DEBUG] [app] Starting request: urn:request:ff2559b8-ae5e-4127-8ada-29fc8b792c8a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:22,065 [243] [DEBUG] [app] Starting request: urn:request:3def240a-7f6b-465f-9a80-0c90af4bf854 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:22,065 [230] [DEBUG] [app] Ending request: urn:request:ff2559b8-ae5e-4127-8ada-29fc8b792c8a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff2559b8-ae5e-4127-8ada-29fc8b792c8a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:22,065 [243] [DEBUG] [app] Ending request: urn:request:3def240a-7f6b-465f-9a80-0c90af4bf854 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3def240a-7f6b-465f-9a80-0c90af4bf854', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:52:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:52:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:22,070 [231] [DEBUG] [app] Starting request: urn:request:4a9d216b-6019-4f9f-89b6-e166fdeece4e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:22,070 [231] [DEBUG] [app] Ending request: urn:request:4a9d216b-6019-4f9f-89b6-e166fdeece4e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a9d216b-6019-4f9f-89b6-e166fdeece4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:22,071 [235] [DEBUG] [app] Starting request: urn:request:98c7c745-b305-427e-b55d-37fea6089158 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:22,071 [235] [DEBUG] [app] Ending request: urn:request:98c7c745-b305-427e-b55d-37fea6089158 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98c7c745-b305-427e-b55d-37fea6089158', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:22,087 [231] [DEBUG] [app] Ending request: urn:request:3189b6c3-3c52-4d81-b12f-3b3ec83e8ea0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3189b6c3-3c52-4d81-b12f-3b3ec83e8ea0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:52:22,088 [236] [DEBUG] [app] Ending request: urn:request:93f70443-850b-4161-be82-fc5832217d17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93f70443-850b-4161-be82-fc5832217d17', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) builder stdout | 2023-01-17 05:52:25,137 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:25,150 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:25,150 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:25,160 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:35.646354+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 05:52:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:35 UTC)" (scheduled at 2023-01-17 05:52:25.646354+00:00) notificationworker stdout | 2023-01-17 05:52:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 25, 647251), True, datetime.datetime(2023, 1, 17, 5, 52, 25, 647251), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:52:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:52:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:35.853396+00:00 (in 6.889168 seconds) exportactionlogsworker stdout | 2023-01-17 05:52:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:28 UTC)" (scheduled at 2023-01-17 05:52:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:52:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:52:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:29,951 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:52:30,002 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:52:30,832 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:52:31,352 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:52:31,355 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:52:31,357 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:52:31,360 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:52:31,362 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:52:31,440 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:52:31,461 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:52:31,549 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:52:31,994 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} expiredappspecifictokenworker stdout | 2023-01-17 05:52:35,137 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} buildlogsarchiver stdout | 2023-01-17 05:52:35,157 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} builder stdout | 2023-01-17 05:52:35,170 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:35,183 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:35,183 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:35,194 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 05:52:35,347 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:52:35,350 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:52:35,352 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:52:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:45.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 05:52:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:45 UTC)" (scheduled at 2023-01-17 05:52:35.646354+00:00) notificationworker stdout | 2023-01-17 05:52:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 35, 647214), True, datetime.datetime(2023, 1, 17, 5, 52, 35, 647214), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:52:35,764 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:52:35,767 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:52:35,769 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:52:35,772 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:52:35,774 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:52:35,777 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:52:35,779 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:52:35,783 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:52:35,785 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:52:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:52:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:28.963662+00:00 (in 53.109791 seconds) exportactionlogsworker stdout | 2023-01-17 05:52:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:35 UTC)" (scheduled at 2023-01-17 05:52:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:52:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:52:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 35, 854268), True, datetime.datetime(2023, 1, 17, 5, 52, 35, 854268), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:52:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:52:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:52:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:52:36,236 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:52:36,371 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:52:36,601 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:52:37,060 [231] [DEBUG] [app] Starting request: urn:request:43b73362-8a99-4fbc-8f4e-9fa7614a07ae (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:37,060 [236] [DEBUG] [app] Starting request: urn:request:40c1a748-d8cb-470d-85b2-8f044dbf3668 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:37,065 [230] [DEBUG] [app] Starting request: urn:request:ba098ca8-9ffb-446d-843f-26a2671707db (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:37,065 [230] [DEBUG] [app] Ending request: urn:request:ba098ca8-9ffb-446d-843f-26a2671707db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ba098ca8-9ffb-446d-843f-26a2671707db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:37,065 [243] [DEBUG] [app] Starting request: urn:request:387f9365-2c33-4c63-93d1-a47cd21ad409 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:37,066 [243] [DEBUG] [app] Ending request: urn:request:387f9365-2c33-4c63-93d1-a47cd21ad409 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:387f9365-2c33-4c63-93d1-a47cd21ad409', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:52:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:52:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:37,069 [235] [DEBUG] [app] Starting request: urn:request:421fdccd-dfb5-4936-98db-c3cc86f47382 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:37,070 [236] [DEBUG] [app] Starting request: urn:request:03d847d4-b3e7-48a6-bd05-6283e87ec8f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:37,070 [235] [DEBUG] [app] Ending request: urn:request:421fdccd-dfb5-4936-98db-c3cc86f47382 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:421fdccd-dfb5-4936-98db-c3cc86f47382', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:37,070 [236] [DEBUG] [app] Ending request: urn:request:03d847d4-b3e7-48a6-bd05-6283e87ec8f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03d847d4-b3e7-48a6-bd05-6283e87ec8f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:52:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:37,087 [236] [DEBUG] [app] Ending request: urn:request:40c1a748-d8cb-470d-85b2-8f044dbf3668 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:40c1a748-d8cb-470d-85b2-8f044dbf3668', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:37,087 [231] [DEBUG] [app] Ending request: urn:request:43b73362-8a99-4fbc-8f4e-9fa7614a07ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43b73362-8a99-4fbc-8f4e-9fa7614a07ae', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:52:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) chunkcleanupworker stdout | 2023-01-17 05:52:37,139 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} securityscanningnotificationworker stdout | 2023-01-17 05:52:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:52:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:37.537407+00:00 (in 0.280348 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:52:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:37 UTC)" (scheduled at 2023-01-17 05:52:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:52:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:52:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 37, 257520), True, datetime.datetime(2023, 1, 17, 5, 52, 37, 257520), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:52:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:52:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:52:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:37 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:52:37,278 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} repositoryactioncounter stdout | 2023-01-17 05:52:37,525 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} securityscanningnotificationworker stdout | 2023-01-17 05:52:37,527 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} securityscanningnotificationworker stdout | 2023-01-17 05:52:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:52:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:37.256652+00:00 (in 59.718846 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:52:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:37 UTC)" (scheduled at 2023-01-17 05:52:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:52:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:52:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:37 UTC)" executed successfully globalpromstats stdout | 2023-01-17 05:52:38,217 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:52:38,733 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:52:38,755 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:52:39,120 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:52:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:52:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:10.738568+00:00 (in 29.999474 seconds) buildlogsarchiver stdout | 2023-01-17 05:52:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:10 UTC)" (scheduled at 2023-01-17 05:52:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:52:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 52, 40, 739518), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:52:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:52:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:52:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:52:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:52:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:42.941536+00:00 (in 1.792854 seconds) namespacegcworker stdout | 2023-01-17 05:52:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:41 UTC)" (scheduled at 2023-01-17 05:52:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:52:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:52:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 41, 149127), True, datetime.datetime(2023, 1, 17, 5, 52, 41, 149127), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:52:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:52:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:52:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:52:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:52:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:41.148245+00:00 (in 58.206194 seconds) namespacegcworker stdout | 2023-01-17 05:52:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:42 UTC)" (scheduled at 2023-01-17 05:52:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:52:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:52:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:53:42 UTC)" executed successfully builder stdout | 2023-01-17 05:52:45,204 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:45,216 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:45,216 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:45,227 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:55.646354+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 05:52:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:55 UTC)" (scheduled at 2023-01-17 05:52:45.646354+00:00) notificationworker stdout | 2023-01-17 05:52:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 45, 647218), True, datetime.datetime(2023, 1, 17, 5, 52, 45, 647218), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:52:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:52:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:52:49.537472+00:00 (in 2.600004 seconds) securityworker stdout | 2023-01-17 05:52:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:16 UTC)" (scheduled at 2023-01-17 05:52:46.936988+00:00) securityworker stdout | 2023-01-17 05:52:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:46,953 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:46,956 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:46,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:52:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:52:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:16.936988+00:00 (in 27.399036 seconds) securityworker stdout | 2023-01-17 05:52:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:19 UTC)" (scheduled at 2023-01-17 05:52:49.537472+00:00) securityworker stdout | 2023-01-17 05:52:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:52:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:52:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:52:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:52:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:52:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:52:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:52:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:52:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:52:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:52:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:52:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:52:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:52:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:52:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:52:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:21.348605+00:00 (in 29.999545 seconds) gcworker stdout | 2023-01-17 05:52:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:21 UTC)" (scheduled at 2023-01-17 05:52:51.348605+00:00) gcworker stdout | 2023-01-17 05:52:51,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 05:52:51,360 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725171360, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:52:51,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:52:51,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:52:51,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:52:52,060 [236] [DEBUG] [app] Starting request: urn:request:f1dad923-6ac9-44df-a9b4-edd8346476b6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:52,060 [235] [DEBUG] [app] Starting request: urn:request:d4e1c15d-f255-44fa-9ec6-fe2749d4afc3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:52:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:52:52,065 [230] [DEBUG] [app] Starting request: urn:request:80f72b2d-52c3-4888-92e2-5277913cb5d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:52,065 [243] [DEBUG] [app] Starting request: urn:request:ebef71b1-6de7-4b29-a71b-d0ea486e9f06 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:52:52,065 [230] [DEBUG] [app] Ending request: urn:request:80f72b2d-52c3-4888-92e2-5277913cb5d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:80f72b2d-52c3-4888-92e2-5277913cb5d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:52:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:52:52,066 [243] [DEBUG] [app] Ending request: urn:request:ebef71b1-6de7-4b29-a71b-d0ea486e9f06 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ebef71b1-6de7-4b29-a71b-d0ea486e9f06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:52:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:52:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:52:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:52:52,070 [233] [DEBUG] [app] Starting request: urn:request:05a4caa0-6ccd-432b-9ec2-11e5b3487539 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:52,070 [235] [DEBUG] [app] Starting request: urn:request:a0ed99c8-3cf1-426d-9c03-26cd78aa5d45 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:52:52,070 [233] [DEBUG] [app] Ending request: urn:request:05a4caa0-6ccd-432b-9ec2-11e5b3487539 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05a4caa0-6ccd-432b-9ec2-11e5b3487539', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:52,070 [235] [DEBUG] [app] Ending request: urn:request:a0ed99c8-3cf1-426d-9c03-26cd78aa5d45 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0ed99c8-3cf1-426d-9c03-26cd78aa5d45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:52:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:52:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:52:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:52:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:52:52,087 [235] [DEBUG] [app] Ending request: urn:request:d4e1c15d-f255-44fa-9ec6-fe2749d4afc3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d4e1c15d-f255-44fa-9ec6-fe2749d4afc3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:52,088 [236] [DEBUG] [app] Ending request: urn:request:f1dad923-6ac9-44df-a9b4-edd8346476b6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1dad923-6ac9-44df-a9b4-edd8346476b6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:52:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:52:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:52:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:52:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:52:55,237 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:52:55,250 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:52:55,250 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:52:55,260 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:52:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:01.636362+00:00 (in 5.989570 seconds) notificationworker stdout | 2023-01-17 05:52:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:05 UTC)" (scheduled at 2023-01-17 05:52:55.646354+00:00) notificationworker stdout | 2023-01-17 05:52:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:52:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 52, 55, 647206), True, datetime.datetime(2023, 1, 17, 5, 52, 55, 647206), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:52:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:52:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:52:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:52:59,967 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:53:00,037 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:53:00,867 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:53:01,359 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:53:01,361 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:53:01,365 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:53:01,367 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:53:01,369 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:53:01,475 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:53:01,492 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:53:01,584 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:53:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:05.646354+00:00 (in 4.009563 seconds) notificationworker stdout | 2023-01-17 05:53:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:01 UTC)" (scheduled at 2023-01-17 05:53:01.636362+00:00) notificationworker stdout | 2023-01-17 05:53:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:53:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:53:02,030 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:53:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:53:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:19.346120+00:00 (in 14.408842 seconds) repositorygcworker stdout | 2023-01-17 05:53:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:04 UTC)" (scheduled at 2023-01-17 05:53:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:53:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:53:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 4, 937702), True, datetime.datetime(2023, 1, 17, 5, 53, 4, 937702), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:53:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:53:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:53:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:04 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:53:05,165 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:53:05,172 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} builder stdout | 2023-01-17 05:53:05,270 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:05,282 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:05,282 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:05,293 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-secscan stdout | 2023-01-17 05:53:05,353 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:53:05,356 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:53:05,358 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:53:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:15.646354+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 05:53:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:15 UTC)" (scheduled at 2023-01-17 05:53:05.646354+00:00) notificationworker stdout | 2023-01-17 05:53:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 5, 647211), True, datetime.datetime(2023, 1, 17, 5, 53, 5, 647211), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:53:05,770 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:53:05,773 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:53:05,780 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:53:05,782 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:53:05,785 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:53:05,787 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:53:05,789 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:53:05,792 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:53:05,795 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:53:06,258 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:53:06,381 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:53:06,636 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:53:07,059 [231] [DEBUG] [app] Starting request: urn:request:59d4b830-474d-44dc-841e-7a2c87aeabf9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:07,060 [236] [DEBUG] [app] Starting request: urn:request:1ceb3663-94d3-4172-acbf-0ac2ac7e8fdd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:07,065 [230] [DEBUG] [app] Starting request: urn:request:15f1479c-81fe-47af-b251-b926111eea60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:07,065 [230] [DEBUG] [app] Ending request: urn:request:15f1479c-81fe-47af-b251-b926111eea60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15f1479c-81fe-47af-b251-b926111eea60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:53:07,065 [243] [DEBUG] [app] Starting request: urn:request:4a3a8c65-5424-4385-878a-3a729c132ce1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:53:07,066 [243] [DEBUG] [app] Ending request: urn:request:4a3a8c65-5424-4385-878a-3a729c132ce1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a3a8c65-5424-4385-878a-3a729c132ce1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:53:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:53:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:07,070 [235] [DEBUG] [app] Starting request: urn:request:92a2b604-3c0c-43d2-90bc-366fff5f9b1e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:07,070 [233] [DEBUG] [app] Starting request: urn:request:d5d70f28-3336-496c-99bc-d3bc29973ae1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:07,070 [235] [DEBUG] [app] Ending request: urn:request:92a2b604-3c0c-43d2-90bc-366fff5f9b1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92a2b604-3c0c-43d2-90bc-366fff5f9b1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:07,071 [233] [DEBUG] [app] Ending request: urn:request:d5d70f28-3336-496c-99bc-d3bc29973ae1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5d70f28-3336-496c-99bc-d3bc29973ae1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:53:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:07,087 [231] [DEBUG] [app] Ending request: urn:request:59d4b830-474d-44dc-841e-7a2c87aeabf9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:59d4b830-474d-44dc-841e-7a2c87aeabf9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:07,087 [236] [DEBUG] [app] Ending request: urn:request:1ceb3663-94d3-4172-acbf-0ac2ac7e8fdd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ceb3663-94d3-4172-acbf-0ac2ac7e8fdd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:53:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) chunkcleanupworker stdout | 2023-01-17 05:53:07,175 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:53:07,303 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:53:07,549 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:53:07,551 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:53:08,231 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:53:08,758 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:53:08,782 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:53:09,132 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:53:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:53:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:40.738568+00:00 (in 29.999390 seconds) buildlogsarchiver stdout | 2023-01-17 05:53:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:40 UTC)" (scheduled at 2023-01-17 05:53:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:53:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 53, 10, 739617), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:53:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:53:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:53:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:40 UTC)" executed successfully builder stdout | 2023-01-17 05:53:15,303 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:15,315 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:15,315 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:15,326 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:25.646354+00:00 (in 9.999570 seconds) notificationworker stdout | 2023-01-17 05:53:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:25 UTC)" (scheduled at 2023-01-17 05:53:15.646354+00:00) notificationworker stdout | 2023-01-17 05:53:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 15, 647198), True, datetime.datetime(2023, 1, 17, 5, 53, 15, 647198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:53:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:19.537472+00:00 (in 2.600002 seconds) securityworker stdout | 2023-01-17 05:53:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:46 UTC)" (scheduled at 2023-01-17 05:53:16.936988+00:00) securityworker stdout | 2023-01-17 05:53:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:53:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:53:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:04.936835+00:00 (in 45.590253 seconds) repositorygcworker stdout | 2023-01-17 05:53:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:19 UTC)" (scheduled at 2023-01-17 05:53:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:53:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:53:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:53:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:46.936988+00:00 (in 27.399032 seconds) securityworker stdout | 2023-01-17 05:53:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:49 UTC)" (scheduled at 2023-01-17 05:53:19.537472+00:00) securityworker stdout | 2023-01-17 05:53:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:53:19,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:53:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:53:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:53:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:53:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:53:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:53:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:53:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:51.348605+00:00 (in 29.999509 seconds) gcworker stdout | 2023-01-17 05:53:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:51 UTC)" (scheduled at 2023-01-17 05:53:21.348605+00:00) gcworker stdout | 2023-01-17 05:53:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725201349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:53:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:53:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:53:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:53:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:53:22,060 [231] [DEBUG] [app] Starting request: urn:request:2282b1c0-e1f9-4c98-ac16-33d095d207e7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:22,060 [235] [DEBUG] [app] Starting request: urn:request:4a473da1-0be5-455e-8355-e116fb24a62b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:22,064 [243] [DEBUG] [app] Starting request: urn:request:b2e3f9c7-8eaf-4321-a1e8-2e94e2717467 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:22,065 [230] [DEBUG] [app] Starting request: urn:request:279d3ca8-d567-4c30-ab3b-686889d92e04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:22,065 [243] [DEBUG] [app] Ending request: urn:request:b2e3f9c7-8eaf-4321-a1e8-2e94e2717467 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2e3f9c7-8eaf-4321-a1e8-2e94e2717467', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:53:22,065 [230] [DEBUG] [app] Ending request: urn:request:279d3ca8-d567-4c30-ab3b-686889d92e04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:279d3ca8-d567-4c30-ab3b-686889d92e04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:53:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:53:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:53:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:22,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:22,070 [233] [DEBUG] [app] Starting request: urn:request:9cc104fb-7309-4c1f-b26e-da41b942bb63 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:22,070 [231] [DEBUG] [app] Starting request: urn:request:ee6206e2-b522-46ed-aaa0-52f3c8c43049 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:22,070 [233] [DEBUG] [app] Ending request: urn:request:9cc104fb-7309-4c1f-b26e-da41b942bb63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9cc104fb-7309-4c1f-b26e-da41b942bb63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:22,070 [231] [DEBUG] [app] Ending request: urn:request:ee6206e2-b522-46ed-aaa0-52f3c8c43049 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee6206e2-b522-46ed-aaa0-52f3c8c43049', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:22,088 [235] [DEBUG] [app] Ending request: urn:request:4a473da1-0be5-455e-8355-e116fb24a62b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4a473da1-0be5-455e-8355-e116fb24a62b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:22,088 [231] [DEBUG] [app] Ending request: urn:request:2282b1c0-e1f9-4c98-ac16-33d095d207e7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2282b1c0-e1f9-4c98-ac16-33d095d207e7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:53:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 05:53:25,336 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:25,349 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:25,349 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:25,360 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:35.646354+00:00 (in 9.999536 seconds) notificationworker stdout | 2023-01-17 05:53:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:35 UTC)" (scheduled at 2023-01-17 05:53:25.646354+00:00) notificationworker stdout | 2023-01-17 05:53:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 25, 647243), True, datetime.datetime(2023, 1, 17, 5, 53, 25, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:53:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:53:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:35.853396+00:00 (in 6.889325 seconds) exportactionlogsworker stdout | 2023-01-17 05:53:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:28 UTC)" (scheduled at 2023-01-17 05:53:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:53:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:53:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:53:29,999 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:53:30,071 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:53:30,903 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:53:31,367 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:53:31,370 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:53:31,373 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:53:31,375 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:53:31,377 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:53:31,480 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:53:31,498 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:53:31,593 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:53:32,038 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} buildlogsarchiver stdout | 2023-01-17 05:53:35,193 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:53:35,197 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:53:35,360 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:53:35,362 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:53:35,364 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} builder stdout | 2023-01-17 05:53:35,370 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:35,382 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:35,382 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:35,393 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:36.635997+00:00 (in 0.989181 seconds) notificationworker stdout | 2023-01-17 05:53:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:45 UTC)" (scheduled at 2023-01-17 05:53:35.646354+00:00) notificationworker stdout | 2023-01-17 05:53:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 35, 647231), True, datetime.datetime(2023, 1, 17, 5, 53, 35, 647231), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:53:35,782 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:53:35,785 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:53:35,790 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:53:35,793 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:53:35,796 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:53:35,798 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:53:35,801 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:53:35,803 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:53:35,806 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:53:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:53:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:28.963662+00:00 (in 53.109846 seconds) exportactionlogsworker stdout | 2023-01-17 05:53:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:35 UTC)" (scheduled at 2023-01-17 05:53:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:53:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:53:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 35, 854197), True, datetime.datetime(2023, 1, 17, 5, 53, 35, 854197), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:53:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:53:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:53:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:53:36,292 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:53:36,392 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:53:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:45.646354+00:00 (in 9.009862 seconds) notificationworker stdout | 2023-01-17 05:53:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:58:36 UTC)" (scheduled at 2023-01-17 05:53:36.635997+00:00) notificationworker stdout | 2023-01-17 05:53:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 53, 36, 636854), 'notification/%']) notificationworker stdout | 2023-01-17 05:53:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 53, 36, 636854), True, datetime.datetime(2023, 1, 17, 5, 53, 36, 636854), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:53:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 53, 36, 636854), True, datetime.datetime(2023, 1, 17, 5, 53, 36, 636854), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 53, 36, 636854), 'notification/%']) notificationworker stdout | 2023-01-17 05:53:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 05:58:36 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:53:36,672 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:53:37,059 [231] [DEBUG] [app] Starting request: urn:request:4187b5d3-e73e-4d10-adab-c9245234611a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:37,059 [236] [DEBUG] [app] Starting request: urn:request:528ab0bd-1589-40b5-ae17-792f6f1a792f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:37,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,060 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:37,064 [230] [DEBUG] [app] Starting request: urn:request:9586d8a9-d181-4a7e-b96b-92b52c273de1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:37,064 [243] [DEBUG] [app] Starting request: urn:request:b11a2d75-7187-4d7f-87dc-a779f019e692 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:37,065 [230] [DEBUG] [app] Ending request: urn:request:9586d8a9-d181-4a7e-b96b-92b52c273de1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9586d8a9-d181-4a7e-b96b-92b52c273de1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:53:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:53:37,065 [243] [DEBUG] [app] Ending request: urn:request:b11a2d75-7187-4d7f-87dc-a779f019e692 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b11a2d75-7187-4d7f-87dc-a779f019e692', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:53:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:53:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:37,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:37,070 [235] [DEBUG] [app] Starting request: urn:request:7e5195c3-cb78-4eb6-9872-92b6d450c7c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:37,070 [236] [DEBUG] [app] Starting request: urn:request:8c28345e-bd8d-4292-88be-dea85fc8345a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:37,071 [235] [DEBUG] [app] Ending request: urn:request:7e5195c3-cb78-4eb6-9872-92b6d450c7c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e5195c3-cb78-4eb6-9872-92b6d450c7c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:37,071 [236] [DEBUG] [app] Ending request: urn:request:8c28345e-bd8d-4292-88be-dea85fc8345a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c28345e-bd8d-4292-88be-dea85fc8345a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:53:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:37,088 [236] [DEBUG] [app] Ending request: urn:request:528ab0bd-1589-40b5-ae17-792f6f1a792f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:528ab0bd-1589-40b5-ae17-792f6f1a792f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:37,088 [231] [DEBUG] [app] Ending request: urn:request:4187b5d3-e73e-4d10-adab-c9245234611a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4187b5d3-e73e-4d10-adab-c9245234611a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:53:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) chunkcleanupworker stdout | 2023-01-17 05:53:37,211 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} securityscanningnotificationworker stdout | 2023-01-17 05:53:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:53:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:37.537407+00:00 (in 0.280352 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:53:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:37 UTC)" (scheduled at 2023-01-17 05:53:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:53:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:53:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 37, 257458), True, datetime.datetime(2023, 1, 17, 5, 53, 37, 257458), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:53:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:53:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:53:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:37 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:53:37,339 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:53:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:53:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:37.256652+00:00 (in 59.718894 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:53:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:37 UTC)" (scheduled at 2023-01-17 05:53:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:53:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:53:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:53:37,585 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:53:37,587 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:53:38,238 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:53:38,770 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:53:38,808 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:53:39,164 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:53:40,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:53:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:10.738568+00:00 (in 29.999417 seconds) buildlogsarchiver stdout | 2023-01-17 05:53:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:10 UTC)" (scheduled at 2023-01-17 05:53:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:53:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 53, 40, 739639), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:53:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:53:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:53:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:53:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:53:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:42.941536+00:00 (in 1.792753 seconds) namespacegcworker stdout | 2023-01-17 05:53:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:41 UTC)" (scheduled at 2023-01-17 05:53:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:53:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:53:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 41, 149206), True, datetime.datetime(2023, 1, 17, 5, 53, 41, 149206), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:53:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:53:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:53:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:53:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:53:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:41.148245+00:00 (in 58.206141 seconds) namespacegcworker stdout | 2023-01-17 05:53:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:42 UTC)" (scheduled at 2023-01-17 05:53:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:53:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:53:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:54:42 UTC)" executed successfully builder stdout | 2023-01-17 05:53:45,393 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:45,406 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:45,406 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:45,417 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:55.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 05:53:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:55 UTC)" (scheduled at 2023-01-17 05:53:45.646354+00:00) notificationworker stdout | 2023-01-17 05:53:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 45, 647197), True, datetime.datetime(2023, 1, 17, 5, 53, 45, 647197), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:45,665 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:45,665 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:45,665 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:53:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:53:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:53:49.537472+00:00 (in 2.600022 seconds) securityworker stdout | 2023-01-17 05:53:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:16 UTC)" (scheduled at 2023-01-17 05:53:46.936988+00:00) securityworker stdout | 2023-01-17 05:53:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:53:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:53:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:16.936988+00:00 (in 27.399030 seconds) securityworker stdout | 2023-01-17 05:53:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:19 UTC)" (scheduled at 2023-01-17 05:53:49.537472+00:00) securityworker stdout | 2023-01-17 05:53:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:53:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:53:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:53:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:53:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:53:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:53:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:53:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:53:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:53:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:53:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:53:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:53:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:53:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:53:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:53:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:21.348605+00:00 (in 29.999505 seconds) gcworker stdout | 2023-01-17 05:53:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:21 UTC)" (scheduled at 2023-01-17 05:53:51.348605+00:00) gcworker stdout | 2023-01-17 05:53:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725231349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:53:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:53:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:53:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:53:52,060 [235] [DEBUG] [app] Starting request: urn:request:0c3db8b2-d5ba-46c1-86c2-a35595acf600 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:52,060 [231] [DEBUG] [app] Starting request: urn:request:cd073b9c-3d84-4437-afbc-c9e456f7b710 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:53:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:53:52,065 [241] [DEBUG] [app] Starting request: urn:request:abd9eade-8fc6-4501-9e48-f0afd7172795 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:52,065 [230] [DEBUG] [app] Starting request: urn:request:bd5f94c3-c122-4ae5-b525-5cba18e1ac46 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:53:52,065 [230] [DEBUG] [app] Ending request: urn:request:bd5f94c3-c122-4ae5-b525-5cba18e1ac46 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd5f94c3-c122-4ae5-b525-5cba18e1ac46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:53:52,065 [241] [DEBUG] [app] Ending request: urn:request:abd9eade-8fc6-4501-9e48-f0afd7172795 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abd9eade-8fc6-4501-9e48-f0afd7172795', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:53:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:53:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:53:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:53:52,070 [236] [DEBUG] [app] Starting request: urn:request:c8f2b303-8916-4f3b-9b46-1eee10a47af9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:52,071 [235] [DEBUG] [app] Starting request: urn:request:f620c7fd-a749-421d-ace9-db163e9ac879 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:53:52,071 [236] [DEBUG] [app] Ending request: urn:request:c8f2b303-8916-4f3b-9b46-1eee10a47af9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c8f2b303-8916-4f3b-9b46-1eee10a47af9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:53:52,071 [235] [DEBUG] [app] Ending request: urn:request:f620c7fd-a749-421d-ace9-db163e9ac879 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f620c7fd-a749-421d-ace9-db163e9ac879', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:53:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:53:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:53:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:53:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:53:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:53:52,088 [231] [DEBUG] [app] Ending request: urn:request:cd073b9c-3d84-4437-afbc-c9e456f7b710 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd073b9c-3d84-4437-afbc-c9e456f7b710', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:52,088 [235] [DEBUG] [app] Ending request: urn:request:0c3db8b2-d5ba-46c1-86c2-a35595acf600 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c3db8b2-d5ba-46c1-86c2-a35595acf600', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:53:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:53:52,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:53:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:53:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:53:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 05:53:55,427 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:53:55,440 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:53:55,440 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:53:55,450 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:53:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:01.636362+00:00 (in 5.989543 seconds) notificationworker stdout | 2023-01-17 05:53:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:05 UTC)" (scheduled at 2023-01-17 05:53:55.646354+00:00) notificationworker stdout | 2023-01-17 05:53:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:53:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 53, 55, 647206), True, datetime.datetime(2023, 1, 17, 5, 53, 55, 647206), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:53:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:53:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:53:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:00,021 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:54:00,086 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:54:00,939 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:54:01,374 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:54:01,377 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:54:01,379 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:54:01,381 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:54:01,384 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:54:01,494 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:54:01,534 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:54:01,628 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} notificationworker stdout | 2023-01-17 05:54:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:05.646354+00:00 (in 4.009542 seconds) notificationworker stdout | 2023-01-17 05:54:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:01 UTC)" (scheduled at 2023-01-17 05:54:01.636362+00:00) notificationworker stdout | 2023-01-17 05:54:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:54:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:01 UTC)" executed successfully securityworker stdout | 2023-01-17 05:54:02,063 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:54:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:54:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:19.346120+00:00 (in 14.408827 seconds) repositorygcworker stdout | 2023-01-17 05:54:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:04 UTC)" (scheduled at 2023-01-17 05:54:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:54:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:54:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 4, 937753), True, datetime.datetime(2023, 1, 17, 5, 54, 4, 937753), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:54:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:54:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:54:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:04 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:54:05,204 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:54:05,231 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:54:05,366 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:54:05,369 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:54:05,371 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} builder stdout | 2023-01-17 05:54:05,453 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:05,466 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:05,466 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:05,477 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:15.646354+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 05:54:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:15 UTC)" (scheduled at 2023-01-17 05:54:05.646354+00:00) notificationworker stdout | 2023-01-17 05:54:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 5, 647174), True, datetime.datetime(2023, 1, 17, 5, 54, 5, 647174), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:54:05,789 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:54:05,792 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:54:05,798 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:54:05,802 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:54:05,805 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:54:05,808 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:54:05,811 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:54:05,814 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:54:05,816 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:54:06,312 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:54:06,422 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:54:06,679 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:54:07,060 [231] [DEBUG] [app] Starting request: urn:request:ca0a22fc-3f8f-42f9-97da-ee05457cf37d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:07,060 [236] [DEBUG] [app] Starting request: urn:request:a34da0cf-a6b8-403e-a0c2-28df807eb78a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:07,065 [242] [DEBUG] [app] Starting request: urn:request:2e650c97-dd44-4e28-beae-691ca702e545 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:07,065 [243] [DEBUG] [app] Starting request: urn:request:5c3b021a-992d-4b9c-9ffb-ed4dc3cd1d75 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:07,066 [242] [DEBUG] [app] Ending request: urn:request:2e650c97-dd44-4e28-beae-691ca702e545 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2e650c97-dd44-4e28-beae-691ca702e545', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:07,066 [243] [DEBUG] [app] Ending request: urn:request:5c3b021a-992d-4b9c-9ffb-ed4dc3cd1d75 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c3b021a-992d-4b9c-9ffb-ed4dc3cd1d75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:54:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:54:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:07,070 [231] [DEBUG] [app] Starting request: urn:request:dbe1c767-9286-4137-b1db-5f41411631b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:07,071 [231] [DEBUG] [app] Ending request: urn:request:dbe1c767-9286-4137-b1db-5f41411631b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dbe1c767-9286-4137-b1db-5f41411631b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:54:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:07,072 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:07,073 [235] [DEBUG] [app] Starting request: urn:request:20a4f9bd-0572-45e8-84a3-1409b247cdfa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:07,073 [235] [DEBUG] [app] Ending request: urn:request:20a4f9bd-0572-45e8-84a3-1409b247cdfa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:20a4f9bd-0572-45e8-84a3-1409b247cdfa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:07,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:07,074 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:07,074 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:07,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:07,087 [236] [DEBUG] [app] Ending request: urn:request:a34da0cf-a6b8-403e-a0c2-28df807eb78a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a34da0cf-a6b8-403e-a0c2-28df807eb78a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:54:07,089 [231] [DEBUG] [app] Ending request: urn:request:ca0a22fc-3f8f-42f9-97da-ee05457cf37d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca0a22fc-3f8f-42f9-97da-ee05457cf37d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:07,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:07,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.032) chunkcleanupworker stdout | 2023-01-17 05:54:07,246 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:54:07,375 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:54:07,593 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:54:07,621 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:54:08,270 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:54:08,789 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:54:08,838 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:54:09,182 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:54:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:54:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:40.738568+00:00 (in 29.999415 seconds) buildlogsarchiver stdout | 2023-01-17 05:54:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:40 UTC)" (scheduled at 2023-01-17 05:54:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:54:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 54, 10, 739545), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:54:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:54:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:54:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:40 UTC)" executed successfully builder stdout | 2023-01-17 05:54:15,487 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:15,499 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:15,499 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:15,510 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:25.646354+00:00 (in 9.999575 seconds) notificationworker stdout | 2023-01-17 05:54:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:25 UTC)" (scheduled at 2023-01-17 05:54:15.646354+00:00) notificationworker stdout | 2023-01-17 05:54:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 15, 647139), True, datetime.datetime(2023, 1, 17, 5, 54, 15, 647139), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:15,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:25 UTC)" executed successfully securityworker stdout | 2023-01-17 05:54:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:19.537472+00:00 (in 2.600014 seconds) securityworker stdout | 2023-01-17 05:54:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:46 UTC)" (scheduled at 2023-01-17 05:54:16.936988+00:00) securityworker stdout | 2023-01-17 05:54:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:54:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:54:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:04.936835+00:00 (in 45.590230 seconds) repositorygcworker stdout | 2023-01-17 05:54:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:19 UTC)" (scheduled at 2023-01-17 05:54:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:54:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:54:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:54:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:46.936988+00:00 (in 27.399017 seconds) securityworker stdout | 2023-01-17 05:54:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:49 UTC)" (scheduled at 2023-01-17 05:54:19.537472+00:00) securityworker stdout | 2023-01-17 05:54:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:54:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:54:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:54:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:54:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:54:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:54:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:54:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:54:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:51.348605+00:00 (in 29.999509 seconds) gcworker stdout | 2023-01-17 05:54:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:51 UTC)" (scheduled at 2023-01-17 05:54:21.348605+00:00) gcworker stdout | 2023-01-17 05:54:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725261349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:54:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:54:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:54:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:54:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:54:22,060 [235] [DEBUG] [app] Starting request: urn:request:9a106c7f-457d-49cd-b4bc-e6f5cc9c723e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:22,060 [231] [DEBUG] [app] Starting request: urn:request:e6777072-ed4f-405b-a4b7-9340660909a0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:22,065 [243] [DEBUG] [app] Starting request: urn:request:146efc22-d30d-48af-a455-e364be0be0b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:22,066 [230] [DEBUG] [app] Starting request: urn:request:774afecf-da39-46d0-9a14-eb97681a0685 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:22,066 [243] [DEBUG] [app] Ending request: urn:request:146efc22-d30d-48af-a455-e364be0be0b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:146efc22-d30d-48af-a455-e364be0be0b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:54:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:54:22,066 [230] [DEBUG] [app] Ending request: urn:request:774afecf-da39-46d0-9a14-eb97681a0685 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:774afecf-da39-46d0-9a14-eb97681a0685', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:54:22,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:54:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:22,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:22,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:22,072 [235] [DEBUG] [app] Starting request: urn:request:61272b5a-4713-459c-8c83-8eb6815a411a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:22,072 [236] [DEBUG] [app] Starting request: urn:request:1482eaff-b91d-4c51-beaa-3ccb8b127b0b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:22,072 [235] [DEBUG] [app] Ending request: urn:request:61272b5a-4713-459c-8c83-8eb6815a411a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61272b5a-4713-459c-8c83-8eb6815a411a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:22,073 [236] [DEBUG] [app] Ending request: urn:request:1482eaff-b91d-4c51-beaa-3ccb8b127b0b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1482eaff-b91d-4c51-beaa-3ccb8b127b0b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:22,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:22,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:54:22,073 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:22,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:22,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:22,081 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:22,081 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:22,090 [235] [DEBUG] [app] Ending request: urn:request:9a106c7f-457d-49cd-b4bc-e6f5cc9c723e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a106c7f-457d-49cd-b4bc-e6f5cc9c723e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:22,090 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:22,090 [231] [DEBUG] [app] Ending request: urn:request:e6777072-ed4f-405b-a4b7-9340660909a0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6777072-ed4f-405b-a4b7-9340660909a0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:22,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:22,090 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 05:54:22,090 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) builder stdout | 2023-01-17 05:54:25,520 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:25,533 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:25,533 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:25,543 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:35.646354+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 05:54:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:35 UTC)" (scheduled at 2023-01-17 05:54:25.646354+00:00) notificationworker stdout | 2023-01-17 05:54:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 25, 647190), True, datetime.datetime(2023, 1, 17, 5, 54, 25, 647190), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:54:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:54:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:35.853396+00:00 (in 6.889258 seconds) exportactionlogsworker stdout | 2023-01-17 05:54:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:28 UTC)" (scheduled at 2023-01-17 05:54:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:54:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:54:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:54:30,038 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:54:30,091 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:54:30,974 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:54:31,380 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:54:31,384 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:54:31,387 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:54:31,389 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:54:31,392 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:54:31,529 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:54:31,541 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:54:31,633 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:54:32,074 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} buildlogsarchiver stdout | 2023-01-17 05:54:35,235 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:54:35,266 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:54:35,373 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:54:35,376 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:54:35,378 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} builder stdout | 2023-01-17 05:54:35,553 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:35,566 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:35,566 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:35,577 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:45.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 05:54:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:45 UTC)" (scheduled at 2023-01-17 05:54:35.646354+00:00) notificationworker stdout | 2023-01-17 05:54:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 35, 647210), True, datetime.datetime(2023, 1, 17, 5, 54, 35, 647210), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:35,657 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:54:35,799 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:54:35,801 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:54:35,804 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:54:35,810 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:54:35,813 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:54:35,817 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:54:35,820 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:54:35,823 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:54:35,825 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:54:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:54:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:28.963662+00:00 (in 53.109808 seconds) exportactionlogsworker stdout | 2023-01-17 05:54:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:35 UTC)" (scheduled at 2023-01-17 05:54:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:54:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:54:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 35, 854226), True, datetime.datetime(2023, 1, 17, 5, 54, 35, 854226), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:54:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:54:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:54:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:54:36,348 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:54:36,440 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:54:36,688 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:54:37,060 [236] [DEBUG] [app] Starting request: urn:request:e1833731-f58a-455b-909b-6a06f808f529 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:37,060 [231] [DEBUG] [app] Starting request: urn:request:4e3fe1e2-939d-4c76-a033-493779d480fe (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:37,065 [230] [DEBUG] [app] Starting request: urn:request:15f52d89-9951-4d35-8435-24b5619b209d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:37,066 [243] [DEBUG] [app] Starting request: urn:request:3ea418e9-edea-4c14-9d6a-178e6a449eb2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:37,066 [230] [DEBUG] [app] Ending request: urn:request:15f52d89-9951-4d35-8435-24b5619b209d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15f52d89-9951-4d35-8435-24b5619b209d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:37,066 [243] [DEBUG] [app] Ending request: urn:request:3ea418e9-edea-4c14-9d6a-178e6a449eb2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ea418e9-edea-4c14-9d6a-178e6a449eb2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:54:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:54:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:37,071 [235] [DEBUG] [app] Starting request: urn:request:d5fb272a-c493-4b62-814c-f77d920ee4c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:37,071 [233] [DEBUG] [app] Starting request: urn:request:110f4005-2418-4c3e-b4b5-2c9196b107bb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:37,071 [235] [DEBUG] [app] Ending request: urn:request:d5fb272a-c493-4b62-814c-f77d920ee4c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5fb272a-c493-4b62-814c-f77d920ee4c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:37,072 [233] [DEBUG] [app] Ending request: urn:request:110f4005-2418-4c3e-b4b5-2c9196b107bb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:110f4005-2418-4c3e-b4b5-2c9196b107bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:37,088 [236] [DEBUG] [app] Ending request: urn:request:e1833731-f58a-455b-909b-6a06f808f529 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1833731-f58a-455b-909b-6a06f808f529', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:37,088 [231] [DEBUG] [app] Ending request: urn:request:4e3fe1e2-939d-4c76-a033-493779d480fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4e3fe1e2-939d-4c76-a033-493779d480fe', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:54:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:54:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:54:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:37.537407+00:00 (in 0.280371 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:54:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:37 UTC)" (scheduled at 2023-01-17 05:54:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:54:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:54:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 37, 257473), True, datetime.datetime(2023, 1, 17, 5, 54, 37, 257473), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:54:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:54:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:54:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:54:37,272 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:54:37,410 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:54:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:54:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:37.256652+00:00 (in 59.718698 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:54:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:37 UTC)" (scheduled at 2023-01-17 05:54:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:54:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:54:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:54:37,628 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:54:37,656 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:54:38,278 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:54:38,796 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:54:38,873 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:54:39,191 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:54:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:54:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:10.738568+00:00 (in 29.999527 seconds) buildlogsarchiver stdout | 2023-01-17 05:54:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:10 UTC)" (scheduled at 2023-01-17 05:54:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:54:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 54, 40, 739479), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:54:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:54:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:54:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:54:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:54:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:42.941536+00:00 (in 1.792796 seconds) namespacegcworker stdout | 2023-01-17 05:54:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:41 UTC)" (scheduled at 2023-01-17 05:54:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:54:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:54:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 41, 149142), True, datetime.datetime(2023, 1, 17, 5, 54, 41, 149142), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:54:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:54:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:54:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:54:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:54:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:41.148245+00:00 (in 58.206246 seconds) namespacegcworker stdout | 2023-01-17 05:54:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:42 UTC)" (scheduled at 2023-01-17 05:54:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:54:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:54:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:55:42 UTC)" executed successfully builder stdout | 2023-01-17 05:54:45,585 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:45,598 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:45,598 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:45,609 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:55.646354+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:54:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:55 UTC)" (scheduled at 2023-01-17 05:54:45.646354+00:00) notificationworker stdout | 2023-01-17 05:54:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 45, 647208), True, datetime.datetime(2023, 1, 17, 5, 54, 45, 647208), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:54:55 UTC)" executed successfully securityworker stdout | 2023-01-17 05:54:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:54:49.537472+00:00 (in 2.599997 seconds) securityworker stdout | 2023-01-17 05:54:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:16 UTC)" (scheduled at 2023-01-17 05:54:46.936988+00:00) securityworker stdout | 2023-01-17 05:54:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:54:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:54:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:16.936988+00:00 (in 27.399051 seconds) securityworker stdout | 2023-01-17 05:54:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:19 UTC)" (scheduled at 2023-01-17 05:54:49.537472+00:00) securityworker stdout | 2023-01-17 05:54:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:54:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:54:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:54:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:54:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:54:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:54:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:54:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:54:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:54:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:54:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:54:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:54:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:54:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:54:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:54:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:21.348605+00:00 (in 29.999504 seconds) gcworker stdout | 2023-01-17 05:54:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:21 UTC)" (scheduled at 2023-01-17 05:54:51.348605+00:00) gcworker stdout | 2023-01-17 05:54:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725291349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:54:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:54:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:54:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:54:52,060 [236] [DEBUG] [app] Starting request: urn:request:dd0c755e-4dd0-4d74-8123-0119920f57f6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:52,060 [231] [DEBUG] [app] Starting request: urn:request:3fdba1a7-eb9e-4ee0-bbc0-0b3c2e12e592 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:54:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:54:52,065 [243] [DEBUG] [app] Starting request: urn:request:62358efc-0365-4fce-9942-7dfdf31d922e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:52,065 [242] [DEBUG] [app] Starting request: urn:request:5b086624-1238-436b-9f96-f4a0212c33d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:54:52,065 [243] [DEBUG] [app] Ending request: urn:request:62358efc-0365-4fce-9942-7dfdf31d922e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:62358efc-0365-4fce-9942-7dfdf31d922e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:54:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:54:52,065 [242] [DEBUG] [app] Ending request: urn:request:5b086624-1238-436b-9f96-f4a0212c33d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5b086624-1238-436b-9f96-f4a0212c33d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:54:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:54:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:54:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:54:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:54:52,069 [235] [DEBUG] [app] Starting request: urn:request:db150c2e-1bd1-4f35-ab07-241ebdf34517 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:52,070 [236] [DEBUG] [app] Starting request: urn:request:4ee50a19-936b-4dc2-a028-6c6818d90084 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:54:52,070 [235] [DEBUG] [app] Ending request: urn:request:db150c2e-1bd1-4f35-ab07-241ebdf34517 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db150c2e-1bd1-4f35-ab07-241ebdf34517', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:52,070 [236] [DEBUG] [app] Ending request: urn:request:4ee50a19-936b-4dc2-a028-6c6818d90084 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4ee50a19-936b-4dc2-a028-6c6818d90084', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:54:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:54:52,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:54:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:52,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:54:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:54:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:54:52,087 [236] [DEBUG] [app] Ending request: urn:request:dd0c755e-4dd0-4d74-8123-0119920f57f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dd0c755e-4dd0-4d74-8123-0119920f57f6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:52,087 [231] [DEBUG] [app] Ending request: urn:request:3fdba1a7-eb9e-4ee0-bbc0-0b3c2e12e592 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3fdba1a7-eb9e-4ee0-bbc0-0b3c2e12e592', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:54:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:54:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 05:54:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:54:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) builder stdout | 2023-01-17 05:54:55,619 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:54:55,632 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:54:55,632 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:54:55,642 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:54:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:01.636362+00:00 (in 5.989603 seconds) notificationworker stdout | 2023-01-17 05:54:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:05 UTC)" (scheduled at 2023-01-17 05:54:55.646354+00:00) notificationworker stdout | 2023-01-17 05:54:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:54:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 54, 55, 647122), True, datetime.datetime(2023, 1, 17, 5, 54, 55, 647122), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:54:55,657 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:54:55,657 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:54:55,657 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:05 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:00,063 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:55:00,118 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:55:01,007 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:55:01,388 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:55:01,391 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:55:01,393 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:55:01,396 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:55:01,399 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:55:01,564 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:55:01,575 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 05:55:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:05.646354+00:00 (in 4.009531 seconds) notificationworker stdout | 2023-01-17 05:55:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:01 UTC)" (scheduled at 2023-01-17 05:55:01.636362+00:00) notificationworker stdout | 2023-01-17 05:55:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:55:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:01 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:55:01,641 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:55:02,091 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:55:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:55:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:19.346120+00:00 (in 14.408846 seconds) repositorygcworker stdout | 2023-01-17 05:55:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:04 UTC)" (scheduled at 2023-01-17 05:55:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:55:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:55:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 4, 937696), True, datetime.datetime(2023, 1, 17, 5, 55, 4, 937696), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:55:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:55:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:55:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:04 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:55:05,254 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:55:05,279 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:55:05,380 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:55:05,382 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:55:05,384 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} builder stdout | 2023-01-17 05:55:05,643 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue notificationworker stdout | 2023-01-17 05:55:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:15.646354+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 05:55:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:15 UTC)" (scheduled at 2023-01-17 05:55:05.646354+00:00) notificationworker stdout | 2023-01-17 05:55:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 5, 647147), True, datetime.datetime(2023, 1, 17, 5, 55, 5, 647147), 0, 'notification/%', 50, 1, 0]) builder stdout | 2023-01-17 05:55:05,655 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:05,655 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics notificationworker stdout | 2023-01-17 05:55:05,657 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:05,657 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:15 UTC)" executed successfully builder stdout | 2023-01-17 05:55:05,666 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:55:05,809 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:55:05,811 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:55:05,813 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:55:05,818 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:55:05,821 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:55:05,824 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:55:05,829 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:55:05,832 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:55:05,834 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:55:06,370 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:55:06,454 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:55:06,724 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:55:07,060 [231] [DEBUG] [app] Starting request: urn:request:1e547ce2-2539-4142-a3d2-b7fe7c38510a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:07,060 [235] [DEBUG] [app] Starting request: urn:request:2b8a4631-3ab1-4f86-84fa-85b52f05e778 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:07,065 [241] [DEBUG] [app] Starting request: urn:request:8ab54c23-eb64-4f53-81b2-769657a1a489 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:07,065 [243] [DEBUG] [app] Starting request: urn:request:386c0e3e-e08b-43c5-80fc-d43eb63c51e4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:07,065 [243] [DEBUG] [app] Ending request: urn:request:386c0e3e-e08b-43c5-80fc-d43eb63c51e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:386c0e3e-e08b-43c5-80fc-d43eb63c51e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:07,066 [241] [DEBUG] [app] Ending request: urn:request:8ab54c23-eb64-4f53-81b2-769657a1a489 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8ab54c23-eb64-4f53-81b2-769657a1a489', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:55:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.003) gunicorn-web stdout | 2023-01-17 05:55:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:07,070 [236] [DEBUG] [app] Starting request: urn:request:32414d6e-1faf-4c11-8576-420d27fafe51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:07,070 [233] [DEBUG] [app] Starting request: urn:request:311da65b-8974-4c46-9fa6-ea07dbd7c23e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:07,071 [233] [DEBUG] [app] Ending request: urn:request:311da65b-8974-4c46-9fa6-ea07dbd7c23e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:311da65b-8974-4c46-9fa6-ea07dbd7c23e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:07,071 [236] [DEBUG] [app] Ending request: urn:request:32414d6e-1faf-4c11-8576-420d27fafe51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32414d6e-1faf-4c11-8576-420d27fafe51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:55:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:07,088 [231] [DEBUG] [app] Ending request: urn:request:1e547ce2-2539-4142-a3d2-b7fe7c38510a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e547ce2-2539-4142-a3d2-b7fe7c38510a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:07,088 [235] [DEBUG] [app] Ending request: urn:request:2b8a4631-3ab1-4f86-84fa-85b52f05e778 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b8a4631-3ab1-4f86-84fa-85b52f05e778', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:55:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) chunkcleanupworker stdout | 2023-01-17 05:55:07,287 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:55:07,420 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:55:07,649 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:55:07,690 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:55:08,313 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:55:08,826 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:55:08,906 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:55:09,221 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:55:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:55:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:40.738568+00:00 (in 29.999482 seconds) buildlogsarchiver stdout | 2023-01-17 05:55:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:40 UTC)" (scheduled at 2023-01-17 05:55:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:55:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 55, 10, 739558), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:55:10,760 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:55:10,760 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:55:10,760 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:25.646354+00:00 (in 9.999567 seconds) notificationworker stdout | 2023-01-17 05:55:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:25 UTC)" (scheduled at 2023-01-17 05:55:15.646354+00:00) notificationworker stdout | 2023-01-17 05:55:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 15, 647171), True, datetime.datetime(2023, 1, 17, 5, 55, 15, 647171), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:25 UTC)" executed successfully builder stdout | 2023-01-17 05:55:15,676 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:15,687 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:15,687 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:15,698 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:19.537472+00:00 (in 2.599992 seconds) securityworker stdout | 2023-01-17 05:55:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:46 UTC)" (scheduled at 2023-01-17 05:55:16.936988+00:00) securityworker stdout | 2023-01-17 05:55:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:55:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:55:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:04.936835+00:00 (in 45.590270 seconds) repositorygcworker stdout | 2023-01-17 05:55:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:19 UTC)" (scheduled at 2023-01-17 05:55:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:55:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:55:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:55:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:46.936988+00:00 (in 27.399021 seconds) securityworker stdout | 2023-01-17 05:55:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:49 UTC)" (scheduled at 2023-01-17 05:55:19.537472+00:00) securityworker stdout | 2023-01-17 05:55:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:55:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:55:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:55:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:55:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:55:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:55:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:55:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:55:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:51.348605+00:00 (in 29.999446 seconds) gcworker stdout | 2023-01-17 05:55:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:51 UTC)" (scheduled at 2023-01-17 05:55:21.348605+00:00) gcworker stdout | 2023-01-17 05:55:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725321349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:55:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:55:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:55:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:55:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:55:22,060 [236] [DEBUG] [app] Starting request: urn:request:5d63f273-9f43-44ed-ad22-267c73008084 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:22,060 [233] [DEBUG] [app] Starting request: urn:request:8bf5e2e7-3968-47e3-a988-7039844709c0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:22,065 [230] [DEBUG] [app] Starting request: urn:request:d4137957-639e-4eb3-ae22-86f06e98171f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:22,065 [243] [DEBUG] [app] Starting request: urn:request:93631dad-e1f5-4c74-93da-7444a47fc552 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:22,065 [230] [DEBUG] [app] Ending request: urn:request:d4137957-639e-4eb3-ae22-86f06e98171f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d4137957-639e-4eb3-ae22-86f06e98171f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:55:22,065 [243] [DEBUG] [app] Ending request: urn:request:93631dad-e1f5-4c74-93da-7444a47fc552 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93631dad-e1f5-4c74-93da-7444a47fc552', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:55:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:55:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:22,070 [231] [DEBUG] [app] Starting request: urn:request:893ea23d-ae67-48ac-9cb7-46a1680d6924 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:22,071 [231] [DEBUG] [app] Ending request: urn:request:893ea23d-ae67-48ac-9cb7-46a1680d6924 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:893ea23d-ae67-48ac-9cb7-46a1680d6924', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:22,071 [235] [DEBUG] [app] Starting request: urn:request:2e65c9dd-6569-43ed-ba1b-7f9b63933c63 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:22,071 [235] [DEBUG] [app] Ending request: urn:request:2e65c9dd-6569-43ed-ba1b-7f9b63933c63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e65c9dd-6569-43ed-ba1b-7f9b63933c63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:55:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:22,088 [236] [DEBUG] [app] Ending request: urn:request:5d63f273-9f43-44ed-ad22-267c73008084 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d63f273-9f43-44ed-ad22-267c73008084', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:55:22,088 [233] [DEBUG] [app] Ending request: urn:request:8bf5e2e7-3968-47e3-a988-7039844709c0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8bf5e2e7-3968-47e3-a988-7039844709c0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:55:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 05:55:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:35.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 05:55:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:35 UTC)" (scheduled at 2023-01-17 05:55:25.646354+00:00) notificationworker stdout | 2023-01-17 05:55:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 25, 647219), True, datetime.datetime(2023, 1, 17, 5, 55, 25, 647219), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:35 UTC)" executed successfully builder stdout | 2023-01-17 05:55:25,708 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:25,720 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:25,720 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:25,730 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:55:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:55:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:35.853396+00:00 (in 6.889308 seconds) exportactionlogsworker stdout | 2023-01-17 05:55:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:28 UTC)" (scheduled at 2023-01-17 05:55:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:55:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:55:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:30,071 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:55:30,153 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:55:31,043 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:55:31,394 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:55:31,398 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:55:31,401 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:55:31,403 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:55:31,405 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:55:31,570 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:55:31,594 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:55:31,656 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:55:32,120 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} buildlogsarchiver stdout | 2023-01-17 05:55:35,284 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:55:35,287 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:55:35,385 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:55:35,388 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:55:35,391 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:55:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:45.646354+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 05:55:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:45 UTC)" (scheduled at 2023-01-17 05:55:35.646354+00:00) notificationworker stdout | 2023-01-17 05:55:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 35, 647199), True, datetime.datetime(2023, 1, 17, 5, 55, 35, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:45 UTC)" executed successfully builder stdout | 2023-01-17 05:55:35,740 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:35,752 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:35,752 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:35,763 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:55:35,818 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:55:35,821 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:55:35,823 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:55:35,825 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:55:35,829 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:55:35,831 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:55:35,835 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:55:35,838 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:55:35,841 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:55:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:55:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:28.963662+00:00 (in 53.109724 seconds) exportactionlogsworker stdout | 2023-01-17 05:55:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:35 UTC)" (scheduled at 2023-01-17 05:55:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:55:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:55:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 35, 854288), True, datetime.datetime(2023, 1, 17, 5, 55, 35, 854288), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:55:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:55:35,864 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:55:35,864 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:35 UTC)" executed successfully storagereplication stdout | 2023-01-17 05:55:36,406 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:55:36,478 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:55:36,760 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:55:37,059 [235] [DEBUG] [app] Starting request: urn:request:10d92cfe-d2ec-4dda-8211-b5a4bca900a8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:37,059 [236] [DEBUG] [app] Starting request: urn:request:d1906481-f5a7-4418-be47-9e0cea585e67 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:37,064 [242] [DEBUG] [app] Starting request: urn:request:8621c643-96a9-4a13-a131-535e341771a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:37,064 [230] [DEBUG] [app] Starting request: urn:request:cf8f6fb2-57c6-46d0-abe5-05a47a29fa8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:37,065 [242] [DEBUG] [app] Ending request: urn:request:8621c643-96a9-4a13-a131-535e341771a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8621c643-96a9-4a13-a131-535e341771a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:37,065 [230] [DEBUG] [app] Ending request: urn:request:cf8f6fb2-57c6-46d0-abe5-05a47a29fa8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf8f6fb2-57c6-46d0-abe5-05a47a29fa8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:55:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:55:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:37,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:37,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:37,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:37,070 [231] [DEBUG] [app] Starting request: urn:request:d3734e79-460d-41e1-84c6-287047a73028 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:37,070 [235] [DEBUG] [app] Starting request: urn:request:7d2ead6e-3994-46af-a737-ab366e37c6e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:37,070 [231] [DEBUG] [app] Ending request: urn:request:d3734e79-460d-41e1-84c6-287047a73028 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3734e79-460d-41e1-84c6-287047a73028', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:37,071 [235] [DEBUG] [app] Ending request: urn:request:7d2ead6e-3994-46af-a737-ab366e37c6e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d2ead6e-3994-46af-a737-ab366e37c6e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:55:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:37,080 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:37,088 [236] [DEBUG] [app] Ending request: urn:request:d1906481-f5a7-4418-be47-9e0cea585e67 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1906481-f5a7-4418-be47-9e0cea585e67', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:55:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:55:37,089 [235] [DEBUG] [app] Ending request: urn:request:10d92cfe-d2ec-4dda-8211-b5a4bca900a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10d92cfe-d2ec-4dda-8211-b5a4bca900a8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:37,089 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 05:55:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:55:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:55:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:37.537407+00:00 (in 0.280327 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:55:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:37 UTC)" (scheduled at 2023-01-17 05:55:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:55:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:55:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 37, 257493), True, datetime.datetime(2023, 1, 17, 5, 55, 37, 257493), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:55:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:55:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:55:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:55:37,312 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:55:37,436 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:55:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:55:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:23.537048+00:00 (in 45.999135 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:55:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:37 UTC)" (scheduled at 2023-01-17 05:55:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:55:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:55:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:55:37,684 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:55:37,726 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:55:38,344 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:55:38,836 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:55:38,933 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:55:39,256 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:55:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:55:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:10.738568+00:00 (in 29.999453 seconds) buildlogsarchiver stdout | 2023-01-17 05:55:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:10 UTC)" (scheduled at 2023-01-17 05:55:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:55:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 55, 40, 739524), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:55:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:55:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:55:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:55:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:55:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:42.941536+00:00 (in 1.792822 seconds) namespacegcworker stdout | 2023-01-17 05:55:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:41 UTC)" (scheduled at 2023-01-17 05:55:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:55:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:55:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 41, 149160), True, datetime.datetime(2023, 1, 17, 5, 55, 41, 149160), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:55:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:55:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:55:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:55:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:55:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:41.148245+00:00 (in 58.206223 seconds) namespacegcworker stdout | 2023-01-17 05:55:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:42 UTC)" (scheduled at 2023-01-17 05:55:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:55:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:55:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:56:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:55:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:55.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:55:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:55 UTC)" (scheduled at 2023-01-17 05:55:45.646354+00:00) notificationworker stdout | 2023-01-17 05:55:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 45, 647186), True, datetime.datetime(2023, 1, 17, 5, 55, 45, 647186), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:55:55 UTC)" executed successfully builder stdout | 2023-01-17 05:55:45,764 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:45,776 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:45,776 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:45,786 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:55:49.537472+00:00 (in 2.600000 seconds) securityworker stdout | 2023-01-17 05:55:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:16 UTC)" (scheduled at 2023-01-17 05:55:46.936988+00:00) securityworker stdout | 2023-01-17 05:55:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:55:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:55:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:16.936988+00:00 (in 27.399065 seconds) securityworker stdout | 2023-01-17 05:55:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:19 UTC)" (scheduled at 2023-01-17 05:55:49.537472+00:00) securityworker stdout | 2023-01-17 05:55:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:55:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:55:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:55:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:55:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:55:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:55:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:55:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:55:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:55:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:55:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:55:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:55:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:55:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:55:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:55:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:21.348605+00:00 (in 29.999548 seconds) gcworker stdout | 2023-01-17 05:55:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:21 UTC)" (scheduled at 2023-01-17 05:55:51.348605+00:00) gcworker stdout | 2023-01-17 05:55:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725351349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:55:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:55:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:55:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:55:52,060 [235] [DEBUG] [app] Starting request: urn:request:80bc31e9-5435-4971-8918-9c0401cfab9e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:52,060 [231] [DEBUG] [app] Starting request: urn:request:d1750c23-77a8-4d21-9386-e112f893a80b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:55:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:55:52,065 [243] [DEBUG] [app] Starting request: urn:request:d3a15fff-c858-4457-b1e1-17df7f19942d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:52,065 [240] [DEBUG] [app] Starting request: urn:request:f7ef51cd-f498-48f2-a4ca-c559b231028c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:55:52,065 [243] [DEBUG] [app] Ending request: urn:request:d3a15fff-c858-4457-b1e1-17df7f19942d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d3a15fff-c858-4457-b1e1-17df7f19942d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:55:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:55:52,066 [240] [DEBUG] [app] Ending request: urn:request:f7ef51cd-f498-48f2-a4ca-c559b231028c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f7ef51cd-f498-48f2-a4ca-c559b231028c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:55:52,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:55:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:55:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:55:52,071 [233] [DEBUG] [app] Starting request: urn:request:425f96e3-b49d-4651-91d5-f90b56ed585a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:52,071 [231] [DEBUG] [app] Starting request: urn:request:6e876d4d-f3dd-4b8f-86cd-8246b48408dd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:55:52,071 [233] [DEBUG] [app] Ending request: urn:request:425f96e3-b49d-4651-91d5-f90b56ed585a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:425f96e3-b49d-4651-91d5-f90b56ed585a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:52,071 [231] [DEBUG] [app] Ending request: urn:request:6e876d4d-f3dd-4b8f-86cd-8246b48408dd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e876d4d-f3dd-4b8f-86cd-8246b48408dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:55:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:55:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:55:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:55:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:55:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:55:52,088 [235] [DEBUG] [app] Ending request: urn:request:80bc31e9-5435-4971-8918-9c0401cfab9e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:80bc31e9-5435-4971-8918-9c0401cfab9e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:52,088 [231] [DEBUG] [app] Ending request: urn:request:d1750c23-77a8-4d21-9386-e112f893a80b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1750c23-77a8-4d21-9386-e112f893a80b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:55:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:55:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:55:52,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:55:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 05:55:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:55:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:01.636362+00:00 (in 5.989548 seconds) notificationworker stdout | 2023-01-17 05:55:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:05 UTC)" (scheduled at 2023-01-17 05:55:55.646354+00:00) notificationworker stdout | 2023-01-17 05:55:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:55:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 55, 55, 647226), True, datetime.datetime(2023, 1, 17, 5, 55, 55, 647226), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:55:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:55:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:55:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:05 UTC)" executed successfully builder stdout | 2023-01-17 05:55:55,788 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:55:55,800 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:55:55,800 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:55:55,810 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:00,077 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:56:00,160 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:56:01,075 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:56:01,401 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:56:01,405 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:56:01,407 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:56:01,409 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:56:01,412 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:56:01,605 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:56:01,629 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} notificationworker stdout | 2023-01-17 05:56:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:05.646354+00:00 (in 4.009547 seconds) notificationworker stdout | 2023-01-17 05:56:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:01 UTC)" (scheduled at 2023-01-17 05:56:01.636362+00:00) notificationworker stdout | 2023-01-17 05:56:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:56:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:01 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:56:01,691 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:56:02,150 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:56:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:56:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:17.345601+00:00 (in 12.408332 seconds) repositorygcworker stdout | 2023-01-17 05:56:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:04 UTC)" (scheduled at 2023-01-17 05:56:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:56:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:56:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 4, 937692), True, datetime.datetime(2023, 1, 17, 5, 56, 4, 937692), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:56:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:56:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:56:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:04 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:56:05,291 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:56:05,315 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:56:05,391 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:56:05,395 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:56:05,398 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:56:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:15.646354+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 05:56:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:15 UTC)" (scheduled at 2023-01-17 05:56:05.646354+00:00) notificationworker stdout | 2023-01-17 05:56:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 5, 647235), True, datetime.datetime(2023, 1, 17, 5, 56, 5, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:15 UTC)" executed successfully builder stdout | 2023-01-17 05:56:05,820 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue gunicorn-registry stdout | 2023-01-17 05:56:05,828 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:56:05,830 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:56:05,832 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} builder stdout | 2023-01-17 05:56:05,833 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:05,833 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-registry stdout | 2023-01-17 05:56:05,835 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:56:05,837 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:56:05,839 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:56:05,843 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} builder stdout | 2023-01-17 05:56:05,844 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 05:56:05,845 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:56:05,848 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 05:56:06,415 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:56:06,488 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:56:06,776 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:56:07,060 [231] [DEBUG] [app] Starting request: urn:request:8b36bb8c-2f10-4362-81f5-f6c6476198bc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:07,060 [236] [DEBUG] [app] Starting request: urn:request:8369eddd-5c28-4883-925d-2dc8b09822b5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:07,065 [242] [DEBUG] [app] Starting request: urn:request:44e3804c-3031-449a-85d6-65d41a9c8631 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:07,065 [243] [DEBUG] [app] Starting request: urn:request:8384e239-9a0d-4edd-ba00-206de0b05453 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:07,066 [242] [DEBUG] [app] Ending request: urn:request:44e3804c-3031-449a-85d6-65d41a9c8631 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44e3804c-3031-449a-85d6-65d41a9c8631', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:07,066 [243] [DEBUG] [app] Ending request: urn:request:8384e239-9a0d-4edd-ba00-206de0b05453 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8384e239-9a0d-4edd-ba00-206de0b05453', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:56:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:56:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:07,070 [235] [DEBUG] [app] Starting request: urn:request:dee5a4db-c8e8-43a5-84b8-3059e82081e0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:07,070 [233] [DEBUG] [app] Starting request: urn:request:5d34324c-8ce5-425f-ba7b-a3c61ee05534 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:07,071 [235] [DEBUG] [app] Ending request: urn:request:dee5a4db-c8e8-43a5-84b8-3059e82081e0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dee5a4db-c8e8-43a5-84b8-3059e82081e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:07,071 [233] [DEBUG] [app] Ending request: urn:request:5d34324c-8ce5-425f-ba7b-a3c61ee05534 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5d34324c-8ce5-425f-ba7b-a3c61ee05534', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:56:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:07,088 [236] [DEBUG] [app] Ending request: urn:request:8369eddd-5c28-4883-925d-2dc8b09822b5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8369eddd-5c28-4883-925d-2dc8b09822b5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:07,088 [231] [DEBUG] [app] Ending request: urn:request:8b36bb8c-2f10-4362-81f5-f6c6476198bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b36bb8c-2f10-4362-81f5-f6c6476198bc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:56:07,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:56:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 05:56:07,348 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:56:07,472 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:56:07,707 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:56:07,740 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:56:08,371 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:56:08,872 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:56:08,953 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:56:09,283 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:56:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:56:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:40.738568+00:00 (in 29.999489 seconds) buildlogsarchiver stdout | 2023-01-17 05:56:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:40 UTC)" (scheduled at 2023-01-17 05:56:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:56:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 56, 10, 739455), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:56:10,752 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:56:10,753 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:56:10,753 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:25.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 05:56:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:25 UTC)" (scheduled at 2023-01-17 05:56:15.646354+00:00) notificationworker stdout | 2023-01-17 05:56:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 15, 647204), True, datetime.datetime(2023, 1, 17, 5, 56, 15, 647204), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:25 UTC)" executed successfully builder stdout | 2023-01-17 05:56:15,844 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:15,856 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:15,856 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:15,867 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:19.537472+00:00 (in 2.600026 seconds) securityworker stdout | 2023-01-17 05:56:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:46 UTC)" (scheduled at 2023-01-17 05:56:16.936988+00:00) securityworker stdout | 2023-01-17 05:56:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:56:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:56:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:19.346120+00:00 (in 2.000041 seconds) repositorygcworker stdout | 2023-01-17 05:56:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:17 UTC)" (scheduled at 2023-01-17 05:56:17.345601+00:00) repositorygcworker stdout | 2023-01-17 05:56:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 56, 17, 346468), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:56:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 17, 346468), True, datetime.datetime(2023, 1, 17, 5, 56, 17, 346468), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:56:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 17, 346468), True, datetime.datetime(2023, 1, 17, 5, 56, 17, 346468), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 5, 56, 17, 346468), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 05:56:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:56:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:56:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:56:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:04.936835+00:00 (in 45.590290 seconds) repositorygcworker stdout | 2023-01-17 05:56:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:19 UTC)" (scheduled at 2023-01-17 05:56:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:56:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:56:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:56:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:46.936988+00:00 (in 27.399026 seconds) securityworker stdout | 2023-01-17 05:56:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:49 UTC)" (scheduled at 2023-01-17 05:56:19.537472+00:00) securityworker stdout | 2023-01-17 05:56:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:56:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:56:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:56:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:56:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:56:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:56:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:56:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:56:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:51.348605+00:00 (in 29.999572 seconds) gcworker stdout | 2023-01-17 05:56:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:51 UTC)" (scheduled at 2023-01-17 05:56:21.348605+00:00) gcworker stdout | 2023-01-17 05:56:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725381349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:56:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:56:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:56:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:56:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:56:22,060 [233] [DEBUG] [app] Starting request: urn:request:7b72829f-0657-403d-baa8-3810f3c5d33b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:22,060 [236] [DEBUG] [app] Starting request: urn:request:c78be920-695c-4dd1-991a-e2b878430290 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:22,065 [230] [DEBUG] [app] Starting request: urn:request:f1ddf4fe-44f4-4497-8cb8-57665aaeaefe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:22,065 [240] [DEBUG] [app] Starting request: urn:request:e4c3c83e-3ae1-4ffa-bc70-f8dbac304d84 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:22,065 [230] [DEBUG] [app] Ending request: urn:request:f1ddf4fe-44f4-4497-8cb8-57665aaeaefe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f1ddf4fe-44f4-4497-8cb8-57665aaeaefe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:56:22,066 [240] [DEBUG] [app] Ending request: urn:request:e4c3c83e-3ae1-4ffa-bc70-f8dbac304d84 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e4c3c83e-3ae1-4ffa-bc70-f8dbac304d84', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:56:22,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:56:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:22,070 [231] [DEBUG] [app] Starting request: urn:request:b0d8a985-4411-4965-b060-e90f3324816d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:22,070 [231] [DEBUG] [app] Ending request: urn:request:b0d8a985-4411-4965-b060-e90f3324816d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b0d8a985-4411-4965-b060-e90f3324816d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:22,070 [235] [DEBUG] [app] Starting request: urn:request:8e8501d9-f7c7-4df1-9e6b-18d99647395f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:22,071 [235] [DEBUG] [app] Ending request: urn:request:8e8501d9-f7c7-4df1-9e6b-18d99647395f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e8501d9-f7c7-4df1-9e6b-18d99647395f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:56:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:22,087 [233] [DEBUG] [app] Ending request: urn:request:7b72829f-0657-403d-baa8-3810f3c5d33b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7b72829f-0657-403d-baa8-3810f3c5d33b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:22,087 [236] [DEBUG] [app] Ending request: urn:request:c78be920-695c-4dd1-991a-e2b878430290 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c78be920-695c-4dd1-991a-e2b878430290', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:56:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 05:56:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:56:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:37.256652+00:00 (in 13.719168 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:56:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:23 UTC)" (scheduled at 2023-01-17 05:56:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:56:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 56, 23, 537838), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:56:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 23, 537838), True, datetime.datetime(2023, 1, 17, 5, 56, 23, 537838), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:56:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 23, 537838), True, datetime.datetime(2023, 1, 17, 5, 56, 23, 537838), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 5, 56, 23, 537838), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 05:56:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:56:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:35.646354+00:00 (in 9.999578 seconds) notificationworker stdout | 2023-01-17 05:56:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:35 UTC)" (scheduled at 2023-01-17 05:56:25.646354+00:00) notificationworker stdout | 2023-01-17 05:56:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 25, 647188), True, datetime.datetime(2023, 1, 17, 5, 56, 25, 647188), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:35 UTC)" executed successfully builder stdout | 2023-01-17 05:56:25,877 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:25,889 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:25,889 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:25,900 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:56:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:56:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:35.853396+00:00 (in 6.889310 seconds) exportactionlogsworker stdout | 2023-01-17 05:56:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:28 UTC)" (scheduled at 2023-01-17 05:56:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:56:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:56:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:30,088 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:56:30,194 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:56:31,091 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:56:31,409 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:56:31,411 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:56:31,414 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:56:31,416 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:56:31,419 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:56:31,640 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:56:31,654 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:56:31,696 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:56:32,186 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} buildlogsarchiver stdout | 2023-01-17 05:56:35,308 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:56:35,338 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:56:35,397 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:56:35,400 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:56:35,404 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:56:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:45.646354+00:00 (in 9.999558 seconds) notificationworker stdout | 2023-01-17 05:56:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:45 UTC)" (scheduled at 2023-01-17 05:56:35.646354+00:00) notificationworker stdout | 2023-01-17 05:56:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 35, 647196), True, datetime.datetime(2023, 1, 17, 5, 56, 35, 647196), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:35,660 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:35,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:35,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:56:35,836 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:56:35,838 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:56:35,840 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:56:35,843 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:56:35,846 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:56:35,848 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:56:35,850 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:56:35,852 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} exportactionlogsworker stdout | 2023-01-17 05:56:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:56:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:54.963413+00:00 (in 19.109569 seconds) exportactionlogsworker stdout | 2023-01-17 05:56:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:35 UTC)" (scheduled at 2023-01-17 05:56:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:56:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:56:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 35, 854192), True, datetime.datetime(2023, 1, 17, 5, 56, 35, 854192), 0, 'exportactionlogs/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:56:35,858 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} exportactionlogsworker stdout | 2023-01-17 05:56:35,866 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:56:35,866 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:56:35,866 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:35 UTC)" executed successfully builder stdout | 2023-01-17 05:56:35,910 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:35,922 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:35,923 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:35,933 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:56:36,439 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:56:36,523 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:56:36,812 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:56:37,060 [233] [DEBUG] [app] Starting request: urn:request:e7f761e1-3050-4213-9f9b-ae4566a91874 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:37,061 [231] [DEBUG] [app] Starting request: urn:request:82018fe4-2f7c-46ee-a147-2a00c448d1f2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:37,063 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,063 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:37,066 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:37,066 [230] [DEBUG] [app] Starting request: urn:request:75a8eff8-cb43-4dad-beb8-423a31c36af3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:37,067 [230] [DEBUG] [app] Ending request: urn:request:75a8eff8-cb43-4dad-beb8-423a31c36af3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:75a8eff8-cb43-4dad-beb8-423a31c36af3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:56:37,067 [242] [DEBUG] [app] Starting request: urn:request:520b843a-dec9-4887-9e33-31b4682aec65 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:37,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:56:37,067 [242] [DEBUG] [app] Ending request: urn:request:520b843a-dec9-4887-9e33-31b4682aec65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:520b843a-dec9-4887-9e33-31b4682aec65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:56:37,068 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:56:37,068 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:37,069 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,070 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:37,072 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:37,072 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:37,073 [236] [DEBUG] [app] Starting request: urn:request:a28e0a67-e694-4fd1-9c00-d8e79466a58b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:37,073 [235] [DEBUG] [app] Starting request: urn:request:7d6ec1fb-40d8-4a5a-b88a-b3d2d247e9c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:37,073 [236] [DEBUG] [app] Ending request: urn:request:a28e0a67-e694-4fd1-9c00-d8e79466a58b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a28e0a67-e694-4fd1-9c00-d8e79466a58b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:37,073 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:37,073 [235] [DEBUG] [app] Ending request: urn:request:7d6ec1fb-40d8-4a5a-b88a-b3d2d247e9c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d6ec1fb-40d8-4a5a-b88a-b3d2d247e9c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:37,074 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:37,074 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:37,074 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:37,074 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:37,074 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:37,081 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:37,081 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:37,091 [233] [DEBUG] [app] Ending request: urn:request:e7f761e1-3050-4213-9f9b-ae4566a91874 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e7f761e1-3050-4213-9f9b-ae4566a91874', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:37,091 [231] [DEBUG] [app] Ending request: urn:request:82018fe4-2f7c-46ee-a147-2a00c448d1f2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:82018fe4-2f7c-46ee-a147-2a00c448d1f2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:37,091 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:37,091 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) gunicorn-web stdout | 2023-01-17 05:56:37,091 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:56:37,091 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:56:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:56:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:37.537407+00:00 (in 0.280337 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:56:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:37 UTC)" (scheduled at 2023-01-17 05:56:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:56:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:56:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 37, 257440), True, datetime.datetime(2023, 1, 17, 5, 56, 37, 257440), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:56:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:56:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:56:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:56:37,384 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:56:37,496 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:56:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:56:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:37.256652+00:00 (in 59.718855 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:56:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:37 UTC)" (scheduled at 2023-01-17 05:56:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:56:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:56:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:56:37,730 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:56:37,775 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:56:38,406 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:56:38,908 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:56:38,988 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:56:39,301 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:56:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:56:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:10.738568+00:00 (in 29.999529 seconds) buildlogsarchiver stdout | 2023-01-17 05:56:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:10 UTC)" (scheduled at 2023-01-17 05:56:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:56:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 56, 40, 739434), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:56:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:56:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:56:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:56:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:56:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:42.941536+00:00 (in 1.792827 seconds) namespacegcworker stdout | 2023-01-17 05:56:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:41 UTC)" (scheduled at 2023-01-17 05:56:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:56:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:56:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 41, 149102), True, datetime.datetime(2023, 1, 17, 5, 56, 41, 149102), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:56:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:56:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:56:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:56:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:56:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:06.941185+00:00 (in 23.999181 seconds) namespacegcworker stdout | 2023-01-17 05:56:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:42 UTC)" (scheduled at 2023-01-17 05:56:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:56:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:56:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:57:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:55.646354+00:00 (in 9.999544 seconds) notificationworker stdout | 2023-01-17 05:56:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:55 UTC)" (scheduled at 2023-01-17 05:56:45.646354+00:00) notificationworker stdout | 2023-01-17 05:56:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 45, 647221), True, datetime.datetime(2023, 1, 17, 5, 56, 45, 647221), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:56:55 UTC)" executed successfully builder stdout | 2023-01-17 05:56:45,943 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:45,956 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:45,956 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:45,966 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:56:49.537472+00:00 (in 2.600030 seconds) securityworker stdout | 2023-01-17 05:56:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:16 UTC)" (scheduled at 2023-01-17 05:56:46.936988+00:00) securityworker stdout | 2023-01-17 05:56:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:56:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:56:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:16.936988+00:00 (in 27.399036 seconds) securityworker stdout | 2023-01-17 05:56:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:19 UTC)" (scheduled at 2023-01-17 05:56:49.537472+00:00) securityworker stdout | 2023-01-17 05:56:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:56:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:56:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:56:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:56:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:56:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:56:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:56:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:56:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:56:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:56:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:56:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:56:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:56:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:56:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:56:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:21.348605+00:00 (in 29.999562 seconds) gcworker stdout | 2023-01-17 05:56:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:21 UTC)" (scheduled at 2023-01-17 05:56:51.348605+00:00) gcworker stdout | 2023-01-17 05:56:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725411349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:56:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:56:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:56:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:56:52,060 [235] [DEBUG] [app] Starting request: urn:request:778ca62b-5792-40d1-81a2-4ff88e14a983 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:52,060 [231] [DEBUG] [app] Starting request: urn:request:6561a2c6-ec05-4cb9-83a4-9402ae8ce9d5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:56:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:56:52,065 [243] [DEBUG] [app] Starting request: urn:request:12ffc483-8cb7-4412-8b38-8cfb73441b48 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:52,065 [242] [DEBUG] [app] Starting request: urn:request:6f37556f-8c60-4742-bc25-60dd1f162b60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:56:52,065 [243] [DEBUG] [app] Ending request: urn:request:12ffc483-8cb7-4412-8b38-8cfb73441b48 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:12ffc483-8cb7-4412-8b38-8cfb73441b48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 05:56:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:56:52,066 [242] [DEBUG] [app] Ending request: urn:request:6f37556f-8c60-4742-bc25-60dd1f162b60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f37556f-8c60-4742-bc25-60dd1f162b60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:56:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:56:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:56:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:56:52,070 [236] [DEBUG] [app] Starting request: urn:request:fa25792f-e17f-463f-9aa1-908bab91f79f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:52,071 [236] [DEBUG] [app] Ending request: urn:request:fa25792f-e17f-463f-9aa1-908bab91f79f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa25792f-e17f-463f-9aa1-908bab91f79f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:56:52,071 [233] [DEBUG] [app] Starting request: urn:request:45f9800c-1726-42ba-9e3b-66856278c5ea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:56:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:52,071 [233] [DEBUG] [app] Ending request: urn:request:45f9800c-1726-42ba-9e3b-66856278c5ea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45f9800c-1726-42ba-9e3b-66856278c5ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:56:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:56:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:56:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:56:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:56:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:56:52,088 [231] [DEBUG] [app] Ending request: urn:request:6561a2c6-ec05-4cb9-83a4-9402ae8ce9d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6561a2c6-ec05-4cb9-83a4-9402ae8ce9d5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:52,088 [235] [DEBUG] [app] Ending request: urn:request:778ca62b-5792-40d1-81a2-4ff88e14a983 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:778ca62b-5792-40d1-81a2-4ff88e14a983', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:56:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:56:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:56:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:56:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) exportactionlogsworker stdout | 2023-01-17 05:56:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:56:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:28.963662+00:00 (in 33.999790 seconds) exportactionlogsworker stdout | 2023-01-17 05:56:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:54 UTC)" (scheduled at 2023-01-17 05:56:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 05:56:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 56, 54, 964302), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:56:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 54, 964302), True, datetime.datetime(2023, 1, 17, 5, 56, 54, 964302), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:56:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 56, 54, 964302), True, datetime.datetime(2023, 1, 17, 5, 56, 54, 964302), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 5, 56, 54, 964302), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 05:56:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:56:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:01:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:56:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:56:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:01.636362+00:00 (in 5.989523 seconds) notificationworker stdout | 2023-01-17 05:56:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:05 UTC)" (scheduled at 2023-01-17 05:56:55.646354+00:00) notificationworker stdout | 2023-01-17 05:56:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:56:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 56, 55, 647258), True, datetime.datetime(2023, 1, 17, 5, 56, 55, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:56:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:56:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:56:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:05 UTC)" executed successfully builder stdout | 2023-01-17 05:56:55,967 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:56:55,979 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:56:55,979 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:56:55,990 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:00,109 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:57:00,229 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:57:01,100 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:57:01,416 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:57:01,419 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:57:01,422 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:57:01,424 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:57:01,426 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:57:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:05.646354+00:00 (in 4.009543 seconds) notificationworker stdout | 2023-01-17 05:57:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:01 UTC)" (scheduled at 2023-01-17 05:57:01.636362+00:00) notificationworker stdout | 2023-01-17 05:57:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:57:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:01 UTC)" executed successfully builder stdout | 2023-01-17 05:57:01,674 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:57:01,689 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:57:01,724 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:57:02,209 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:57:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:57:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:19.346120+00:00 (in 14.408913 seconds) repositorygcworker stdout | 2023-01-17 05:57:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:04 UTC)" (scheduled at 2023-01-17 05:57:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:57:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:57:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 4, 937633), True, datetime.datetime(2023, 1, 17, 5, 57, 4, 937633), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:57:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:57:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:57:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:04 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:57:05,339 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:57:05,374 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:57:05,403 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:57:05,406 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:57:05,409 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:57:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:15.646354+00:00 (in 9.999544 seconds) notificationworker stdout | 2023-01-17 05:57:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:15 UTC)" (scheduled at 2023-01-17 05:57:05.646354+00:00) notificationworker stdout | 2023-01-17 05:57:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 5, 647335), True, datetime.datetime(2023, 1, 17, 5, 57, 5, 647335), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:57:05,843 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:57:05,845 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:57:05,848 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:57:05,851 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:57:05,853 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:57:05,856 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:57:05,858 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:57:05,862 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:57:05,867 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 05:57:06,000 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:06,012 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:06,012 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:06,023 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:57:06,475 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:57:06,527 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:57:06,847 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} namespacegcworker stdout | 2023-01-17 05:57:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:57:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:41.148245+00:00 (in 34.206571 seconds) namespacegcworker stdout | 2023-01-17 05:57:06,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:02:06 UTC)" (scheduled at 2023-01-17 05:57:06.941185+00:00) namespacegcworker stdout | 2023-01-17 05:57:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 57, 6, 942119), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:57:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 57, 6, 942119), True, datetime.datetime(2023, 1, 17, 5, 57, 6, 942119), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:57:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 57, 6, 942119), True, datetime.datetime(2023, 1, 17, 5, 57, 6, 942119), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 5, 57, 6, 942119), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 05:57:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:57:06,961 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:02:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:57:07,060 [231] [DEBUG] [app] Starting request: urn:request:c088dfd3-82d3-43c4-9968-9bd65c2a5428 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:07,060 [236] [DEBUG] [app] Starting request: urn:request:37b67f70-0605-4132-9ac7-a41d8bb91965 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:07,065 [243] [DEBUG] [app] Starting request: urn:request:5c718933-79f3-473c-9583-644f3b398622 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:07,065 [242] [DEBUG] [app] Starting request: urn:request:c2c74b27-0839-4466-bfce-67ac5a232201 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:07,065 [243] [DEBUG] [app] Ending request: urn:request:5c718933-79f3-473c-9583-644f3b398622 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c718933-79f3-473c-9583-644f3b398622', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:07,066 [242] [DEBUG] [app] Ending request: urn:request:c2c74b27-0839-4466-bfce-67ac5a232201 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2c74b27-0839-4466-bfce-67ac5a232201', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:57:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:57:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:07,071 [235] [DEBUG] [app] Starting request: urn:request:c26d0229-87ec-40b9-9b23-0ce035a347e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:07,071 [233] [DEBUG] [app] Starting request: urn:request:b096ecf0-db25-4488-ba43-ddbf457b9b8c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:07,071 [235] [DEBUG] [app] Ending request: urn:request:c26d0229-87ec-40b9-9b23-0ce035a347e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c26d0229-87ec-40b9-9b23-0ce035a347e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:57:07,071 [233] [DEBUG] [app] Ending request: urn:request:b096ecf0-db25-4488-ba43-ddbf457b9b8c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b096ecf0-db25-4488-ba43-ddbf457b9b8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:57:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:07,088 [231] [DEBUG] [app] Ending request: urn:request:c088dfd3-82d3-43c4-9968-9bd65c2a5428 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c088dfd3-82d3-43c4-9968-9bd65c2a5428', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:07,088 [236] [DEBUG] [app] Ending request: urn:request:37b67f70-0605-4132-9ac7-a41d8bb91965 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37b67f70-0605-4132-9ac7-a41d8bb91965', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:57:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 05:57:07,399 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:57:07,529 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:57:07,745 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:57:07,799 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:57:08,442 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:57:08,940 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:57:09,001 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:57:09,317 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:57:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:57:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:40.738568+00:00 (in 29.999529 seconds) buildlogsarchiver stdout | 2023-01-17 05:57:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:40 UTC)" (scheduled at 2023-01-17 05:57:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:57:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 57, 10, 739385), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:57:10,762 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:57:10,762 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:57:10,763 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:57:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:25.646354+00:00 (in 9.999504 seconds) notificationworker stdout | 2023-01-17 05:57:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:25 UTC)" (scheduled at 2023-01-17 05:57:15.646354+00:00) notificationworker stdout | 2023-01-17 05:57:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 15, 647281), True, datetime.datetime(2023, 1, 17, 5, 57, 15, 647281), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:25 UTC)" executed successfully builder stdout | 2023-01-17 05:57:16,033 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:16,046 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:16,046 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:16,056 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:19.537472+00:00 (in 2.599985 seconds) securityworker stdout | 2023-01-17 05:57:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:46 UTC)" (scheduled at 2023-01-17 05:57:16.936988+00:00) securityworker stdout | 2023-01-17 05:57:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:57:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:57:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:04.936835+00:00 (in 45.590240 seconds) repositorygcworker stdout | 2023-01-17 05:57:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:19 UTC)" (scheduled at 2023-01-17 05:57:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:57:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:57:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:57:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:46.936988+00:00 (in 27.399028 seconds) securityworker stdout | 2023-01-17 05:57:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:49 UTC)" (scheduled at 2023-01-17 05:57:19.537472+00:00) securityworker stdout | 2023-01-17 05:57:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:57:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:57:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:57:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:57:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:57:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:57:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:57:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:57:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:51.348605+00:00 (in 29.999572 seconds) gcworker stdout | 2023-01-17 05:57:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:51 UTC)" (scheduled at 2023-01-17 05:57:21.348605+00:00) gcworker stdout | 2023-01-17 05:57:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725441349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:57:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:57:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:57:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:57:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:57:22,060 [236] [DEBUG] [app] Starting request: urn:request:938c518d-a187-4f6e-afb8-b5d621933ed6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:22,060 [231] [DEBUG] [app] Starting request: urn:request:3dd87226-93c7-412a-8e26-b2b5bfc6c67f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:22,065 [230] [DEBUG] [app] Starting request: urn:request:15ff06a8-5c32-49f3-a007-a0544e31f3c4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:22,065 [230] [DEBUG] [app] Ending request: urn:request:15ff06a8-5c32-49f3-a007-a0544e31f3c4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15ff06a8-5c32-49f3-a007-a0544e31f3c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:22,065 [241] [DEBUG] [app] Starting request: urn:request:3e1fd262-80ac-4e55-8c5c-c4540d81c627 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:57:22,066 [241] [DEBUG] [app] Ending request: urn:request:3e1fd262-80ac-4e55-8c5c-c4540d81c627 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e1fd262-80ac-4e55-8c5c-c4540d81c627', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:57:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:57:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:22,070 [235] [DEBUG] [app] Starting request: urn:request:36e3cb8d-5a27-41b2-aef1-74999527bd98 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:22,071 [235] [DEBUG] [app] Ending request: urn:request:36e3cb8d-5a27-41b2-aef1-74999527bd98 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36e3cb8d-5a27-41b2-aef1-74999527bd98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:22,071 [233] [DEBUG] [app] Starting request: urn:request:55f81ca2-60cd-417e-8641-07b8f5830819 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:57:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:22,071 [233] [DEBUG] [app] Ending request: urn:request:55f81ca2-60cd-417e-8641-07b8f5830819 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55f81ca2-60cd-417e-8641-07b8f5830819', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:57:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:22,088 [231] [DEBUG] [app] Ending request: urn:request:3dd87226-93c7-412a-8e26-b2b5bfc6c67f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3dd87226-93c7-412a-8e26-b2b5bfc6c67f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:22,088 [236] [DEBUG] [app] Ending request: urn:request:938c518d-a187-4f6e-afb8-b5d621933ed6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:938c518d-a187-4f6e-afb8-b5d621933ed6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:22,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:57:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:57:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:57:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:35.646354+00:00 (in 9.999501 seconds) notificationworker stdout | 2023-01-17 05:57:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:35 UTC)" (scheduled at 2023-01-17 05:57:25.646354+00:00) notificationworker stdout | 2023-01-17 05:57:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 25, 647341), True, datetime.datetime(2023, 1, 17, 5, 57, 25, 647341), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:35 UTC)" executed successfully builder stdout | 2023-01-17 05:57:26,066 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:26,079 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:26,079 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:26,089 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:57:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:57:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:35.853396+00:00 (in 6.889204 seconds) exportactionlogsworker stdout | 2023-01-17 05:57:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:28 UTC)" (scheduled at 2023-01-17 05:57:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:57:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:57:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:57:30,145 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:57:30,244 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:57:31,136 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:57:31,424 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:57:31,426 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:57:31,429 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:57:31,431 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:57:31,434 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:57:31,684 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:57:31,723 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:57:31,759 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:57:32,226 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} buildlogsarchiver stdout | 2023-01-17 05:57:35,371 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:57:35,409 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} expiredappspecifictokenworker stdout | 2023-01-17 05:57:35,411 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-secscan stdout | 2023-01-17 05:57:35,413 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:57:35,415 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 05:57:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:45.646354+00:00 (in 9.999570 seconds) notificationworker stdout | 2023-01-17 05:57:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:45 UTC)" (scheduled at 2023-01-17 05:57:35.646354+00:00) notificationworker stdout | 2023-01-17 05:57:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 35, 647175), True, datetime.datetime(2023, 1, 17, 5, 57, 35, 647175), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:45 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:57:35,849 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} exportactionlogsworker stdout | 2023-01-17 05:57:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:57:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:28.963662+00:00 (in 53.109739 seconds) exportactionlogsworker stdout | 2023-01-17 05:57:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:35 UTC)" (scheduled at 2023-01-17 05:57:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:57:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:57:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 35, 854352), True, datetime.datetime(2023, 1, 17, 5, 57, 35, 854352), 0, 'exportactionlogs/%', 50, 1, 0]) gunicorn-registry stdout | 2023-01-17 05:57:35,856 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:57:35,859 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:57:35,861 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:57:35,863 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} exportactionlogsworker stdout | 2023-01-17 05:57:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:57:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:57:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:57:35,867 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:57:35,869 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:57:35,872 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:57:35,876 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 05:57:36,100 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:36,112 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:36,112 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:36,122 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:57:36,511 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:57:36,538 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:57:36,873 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:57:37,060 [235] [DEBUG] [app] Starting request: urn:request:91852025-c12d-4fa8-89f8-cf51648e2087 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:37,060 [236] [DEBUG] [app] Starting request: urn:request:cb7a29aa-d511-4f94-9476-0e927acc3287 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:37,065 [243] [DEBUG] [app] Starting request: urn:request:933d7fc4-b414-4678-a1c3-af98516d582b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:37,066 [242] [DEBUG] [app] Starting request: urn:request:6415c88f-d518-4254-9f2e-89a61ab49cdc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:37,066 [243] [DEBUG] [app] Ending request: urn:request:933d7fc4-b414-4678-a1c3-af98516d582b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:933d7fc4-b414-4678-a1c3-af98516d582b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:57:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:57:37,066 [242] [DEBUG] [app] Ending request: urn:request:6415c88f-d518-4254-9f2e-89a61ab49cdc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6415c88f-d518-4254-9f2e-89a61ab49cdc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:57:37,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:57:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:37,071 [231] [DEBUG] [app] Starting request: urn:request:4b1801a2-4215-4be8-8715-188bd0463add (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:37,071 [235] [DEBUG] [app] Starting request: urn:request:1611b25c-cd4f-44bc-97e4-627003f1db0e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:37,071 [231] [DEBUG] [app] Ending request: urn:request:4b1801a2-4215-4be8-8715-188bd0463add (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b1801a2-4215-4be8-8715-188bd0463add', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:37,071 [235] [DEBUG] [app] Ending request: urn:request:1611b25c-cd4f-44bc-97e4-627003f1db0e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1611b25c-cd4f-44bc-97e4-627003f1db0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:57:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:37,089 [235] [DEBUG] [app] Ending request: urn:request:91852025-c12d-4fa8-89f8-cf51648e2087 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:91852025-c12d-4fa8-89f8-cf51648e2087', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:37,089 [236] [DEBUG] [app] Ending request: urn:request:cb7a29aa-d511-4f94-9476-0e927acc3287 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb7a29aa-d511-4f94-9476-0e927acc3287', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:37,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:37,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 05:57:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:57:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:57:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:37.537407+00:00 (in 0.280353 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:57:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:37 UTC)" (scheduled at 2023-01-17 05:57:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:57:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:57:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 37, 257423), True, datetime.datetime(2023, 1, 17, 5, 57, 37, 257423), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:57:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:57:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:57:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:57:37,414 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} securityscanningnotificationworker stdout | 2023-01-17 05:57:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:57:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:37.256652+00:00 (in 59.718850 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:57:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:37 UTC)" (scheduled at 2023-01-17 05:57:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:57:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:57:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:37 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:57:37,564 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:57:37,754 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:57:37,834 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:57:38,448 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:57:38,976 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:57:39,036 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:57:39,349 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:57:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:57:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:10.738568+00:00 (in 29.999550 seconds) buildlogsarchiver stdout | 2023-01-17 05:57:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:10 UTC)" (scheduled at 2023-01-17 05:57:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:57:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 57, 40, 739367), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:57:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:57:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:57:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:57:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:57:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:42.941536+00:00 (in 1.792728 seconds) namespacegcworker stdout | 2023-01-17 05:57:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:41 UTC)" (scheduled at 2023-01-17 05:57:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:57:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:57:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 41, 149239), True, datetime.datetime(2023, 1, 17, 5, 57, 41, 149239), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:57:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:57:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:57:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:57:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:57:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:41.148245+00:00 (in 58.206216 seconds) namespacegcworker stdout | 2023-01-17 05:57:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:42 UTC)" (scheduled at 2023-01-17 05:57:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:57:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:57:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:58:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:57:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:55.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 05:57:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:55 UTC)" (scheduled at 2023-01-17 05:57:45.646354+00:00) notificationworker stdout | 2023-01-17 05:57:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 45, 647232), True, datetime.datetime(2023, 1, 17, 5, 57, 45, 647232), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:57:55 UTC)" executed successfully builder stdout | 2023-01-17 05:57:46,133 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:46,145 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:46,145 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:46,156 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:57:49.537472+00:00 (in 2.600020 seconds) securityworker stdout | 2023-01-17 05:57:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:16 UTC)" (scheduled at 2023-01-17 05:57:46.936988+00:00) securityworker stdout | 2023-01-17 05:57:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:57:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:57:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:16.936988+00:00 (in 27.399082 seconds) securityworker stdout | 2023-01-17 05:57:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:19 UTC)" (scheduled at 2023-01-17 05:57:49.537472+00:00) securityworker stdout | 2023-01-17 05:57:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:57:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:57:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:57:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:57:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:57:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:57:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:57:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:57:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:57:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:57:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:57:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:57:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:57:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:57:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:57:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:21.348605+00:00 (in 29.999556 seconds) gcworker stdout | 2023-01-17 05:57:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:21 UTC)" (scheduled at 2023-01-17 05:57:51.348605+00:00) gcworker stdout | 2023-01-17 05:57:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725471349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:57:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:57:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:57:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:57:52,060 [231] [DEBUG] [app] Starting request: urn:request:8dd5af31-9d60-4034-96ba-c77265e819d2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:52,060 [236] [DEBUG] [app] Starting request: urn:request:5d460cc2-1afc-45bf-a403-8ef01badf076 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:57:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:57:52,065 [230] [DEBUG] [app] Starting request: urn:request:6a497cf6-82d3-420f-8952-ef0ec468999b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:52,065 [242] [DEBUG] [app] Starting request: urn:request:c068865f-45fa-41a9-8ace-d58462fa17e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:57:52,065 [230] [DEBUG] [app] Ending request: urn:request:6a497cf6-82d3-420f-8952-ef0ec468999b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6a497cf6-82d3-420f-8952-ef0ec468999b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:52,066 [242] [DEBUG] [app] Ending request: urn:request:c068865f-45fa-41a9-8ace-d58462fa17e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c068865f-45fa-41a9-8ace-d58462fa17e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:57:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:57:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:57:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:57:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:57:52,070 [235] [DEBUG] [app] Starting request: urn:request:f9459f36-7263-4e9b-8292-df96b8fd435d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:52,070 [231] [DEBUG] [app] Starting request: urn:request:89cb5184-8924-4475-9976-f22c5ea78f2b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:57:52,071 [235] [DEBUG] [app] Ending request: urn:request:f9459f36-7263-4e9b-8292-df96b8fd435d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9459f36-7263-4e9b-8292-df96b8fd435d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:57:52,071 [231] [DEBUG] [app] Ending request: urn:request:89cb5184-8924-4475-9976-f22c5ea78f2b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89cb5184-8924-4475-9976-f22c5ea78f2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:57:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:57:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:57:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:57:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:57:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:57:52,087 [231] [DEBUG] [app] Ending request: urn:request:8dd5af31-9d60-4034-96ba-c77265e819d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8dd5af31-9d60-4034-96ba-c77265e819d2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:52,087 [236] [DEBUG] [app] Ending request: urn:request:5d460cc2-1afc-45bf-a403-8ef01badf076 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d460cc2-1afc-45bf-a403-8ef01badf076', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:57:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:57:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 05:57:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:57:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:57:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:57:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:01.636362+00:00 (in 5.989522 seconds) notificationworker stdout | 2023-01-17 05:57:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:05 UTC)" (scheduled at 2023-01-17 05:57:55.646354+00:00) notificationworker stdout | 2023-01-17 05:57:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:57:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 57, 55, 647299), True, datetime.datetime(2023, 1, 17, 5, 57, 55, 647299), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:57:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:57:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:57:55,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:05 UTC)" executed successfully builder stdout | 2023-01-17 05:57:56,166 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:57:56,178 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:57:56,178 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:57:56,189 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:00,158 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:58:00,279 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:58:01,147 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:58:01,431 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:58:01,434 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:58:01,436 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:58:01,438 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:58:01,441 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:58:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:05.646354+00:00 (in 4.009511 seconds) notificationworker stdout | 2023-01-17 05:58:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:01 UTC)" (scheduled at 2023-01-17 05:58:01.636362+00:00) notificationworker stdout | 2023-01-17 05:58:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:58:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:01 UTC)" executed successfully builder stdout | 2023-01-17 05:58:01,696 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:58:01,730 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:58:01,794 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:58:02,262 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:58:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:58:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:19.346120+00:00 (in 14.408826 seconds) repositorygcworker stdout | 2023-01-17 05:58:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:04 UTC)" (scheduled at 2023-01-17 05:58:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:58:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:58:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 4, 937750), True, datetime.datetime(2023, 1, 17, 5, 58, 4, 937750), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:58:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:58:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:58:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:04 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 05:58:05,393 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:58:05,414 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:58:05,420 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:58:05,423 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} expiredappspecifictokenworker stdout | 2023-01-17 05:58:05,445 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:58:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:15.646354+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:58:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:15 UTC)" (scheduled at 2023-01-17 05:58:05.646354+00:00) notificationworker stdout | 2023-01-17 05:58:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 5, 647219), True, datetime.datetime(2023, 1, 17, 5, 58, 5, 647219), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:58:05,860 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:58:05,866 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:58:05,868 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:58:05,870 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:58:05,873 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:58:05,878 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:58:05,881 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:58:05,883 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:58:05,885 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 05:58:06,199 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:06,212 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:06,212 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:06,222 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:58:06,547 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:58:06,572 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:58:06,908 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:58:07,060 [231] [DEBUG] [app] Starting request: urn:request:ee0a2af3-91b8-4d74-97f3-ba2375862f7b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:07,060 [236] [DEBUG] [app] Starting request: urn:request:db61b117-728b-44b6-a347-b6b457d146d5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:07,066 [241] [DEBUG] [app] Starting request: urn:request:09025be8-83ab-41c6-9009-629af8b11227 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:07,066 [240] [DEBUG] [app] Starting request: urn:request:3380cd15-3c53-42e5-a214-569d77c93d97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:07,066 [241] [DEBUG] [app] Ending request: urn:request:09025be8-83ab-41c6-9009-629af8b11227 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09025be8-83ab-41c6-9009-629af8b11227', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:58:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:58:07,067 [240] [DEBUG] [app] Ending request: urn:request:3380cd15-3c53-42e5-a214-569d77c93d97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3380cd15-3c53-42e5-a214-569d77c93d97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:07,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 05:58:07,067 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:07,071 [235] [DEBUG] [app] Starting request: urn:request:9003d7bb-d400-4a00-a548-2fdb3a295080 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:07,071 [236] [DEBUG] [app] Starting request: urn:request:33143ad3-de11-42b2-a5ae-f499b11645bf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:07,071 [235] [DEBUG] [app] Ending request: urn:request:9003d7bb-d400-4a00-a548-2fdb3a295080 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9003d7bb-d400-4a00-a548-2fdb3a295080', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:07,072 [236] [DEBUG] [app] Ending request: urn:request:33143ad3-de11-42b2-a5ae-f499b11645bf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:33143ad3-de11-42b2-a5ae-f499b11645bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:07,088 [236] [DEBUG] [app] Ending request: urn:request:db61b117-728b-44b6-a347-b6b457d146d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db61b117-728b-44b6-a347-b6b457d146d5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:07,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:07,089 [231] [DEBUG] [app] Ending request: urn:request:ee0a2af3-91b8-4d74-97f3-ba2375862f7b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee0a2af3-91b8-4d74-97f3-ba2375862f7b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:07,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:58:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) chunkcleanupworker stdout | 2023-01-17 05:58:07,450 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:58:07,573 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:58:07,788 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:58:07,862 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:58:08,484 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:58:09,001 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:58:09,064 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:58:09,360 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:58:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:58:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:40.738568+00:00 (in 29.999574 seconds) buildlogsarchiver stdout | 2023-01-17 05:58:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:40 UTC)" (scheduled at 2023-01-17 05:58:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:58:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 58, 10, 739342), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:58:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:58:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:58:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:58:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:25.646354+00:00 (in 9.999586 seconds) notificationworker stdout | 2023-01-17 05:58:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:25 UTC)" (scheduled at 2023-01-17 05:58:15.646354+00:00) notificationworker stdout | 2023-01-17 05:58:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 15, 647164), True, datetime.datetime(2023, 1, 17, 5, 58, 15, 647164), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:25 UTC)" executed successfully builder stdout | 2023-01-17 05:58:16,222 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:16,234 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:16,235 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:16,245 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:19.537472+00:00 (in 2.600044 seconds) securityworker stdout | 2023-01-17 05:58:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:46 UTC)" (scheduled at 2023-01-17 05:58:16.936988+00:00) securityworker stdout | 2023-01-17 05:58:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:58:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:58:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:04.936835+00:00 (in 45.590191 seconds) repositorygcworker stdout | 2023-01-17 05:58:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:19 UTC)" (scheduled at 2023-01-17 05:58:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:58:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:58:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:58:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:46.936988+00:00 (in 27.399045 seconds) securityworker stdout | 2023-01-17 05:58:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:49 UTC)" (scheduled at 2023-01-17 05:58:19.537472+00:00) securityworker stdout | 2023-01-17 05:58:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:58:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:58:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:58:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:58:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:58:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:58:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:58:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:58:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:51.348605+00:00 (in 29.999535 seconds) gcworker stdout | 2023-01-17 05:58:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:51 UTC)" (scheduled at 2023-01-17 05:58:21.348605+00:00) gcworker stdout | 2023-01-17 05:58:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725501349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:58:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:58:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:58:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:58:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:58:22,060 [236] [DEBUG] [app] Starting request: urn:request:6ad98d6d-13cb-49a0-8d93-9659c177adbd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:22,060 [235] [DEBUG] [app] Starting request: urn:request:09bdf91b-aa79-4373-a7d6-77b39aa54533 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:22,064 [230] [DEBUG] [app] Starting request: urn:request:11f25c38-b349-4cb3-8212-3212906b5f02 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:22,065 [230] [DEBUG] [app] Ending request: urn:request:11f25c38-b349-4cb3-8212-3212906b5f02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11f25c38-b349-4cb3-8212-3212906b5f02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:22,065 [241] [DEBUG] [app] Starting request: urn:request:96c7b28c-ea3a-4e91-b47c-11624ecf3cdd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:58:22,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:58:22,065 [241] [DEBUG] [app] Ending request: urn:request:96c7b28c-ea3a-4e91-b47c-11624ecf3cdd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96c7b28c-ea3a-4e91-b47c-11624ecf3cdd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:22,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:58:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:22,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:22,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:22,069 [231] [DEBUG] [app] Starting request: urn:request:c0314a49-c65e-4295-ba18-def9f591cbc4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:22,070 [231] [DEBUG] [app] Ending request: urn:request:c0314a49-c65e-4295-ba18-def9f591cbc4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c0314a49-c65e-4295-ba18-def9f591cbc4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:22,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:58:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:22,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:22,071 [233] [DEBUG] [app] Starting request: urn:request:8f4046b0-5a53-498c-83bf-a35c91a0e5a0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:22,072 [233] [DEBUG] [app] Ending request: urn:request:8f4046b0-5a53-498c-83bf-a35c91a0e5a0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8f4046b0-5a53-498c-83bf-a35c91a0e5a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:58:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:22,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:22,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:22,088 [235] [DEBUG] [app] Ending request: urn:request:09bdf91b-aa79-4373-a7d6-77b39aa54533 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09bdf91b-aa79-4373-a7d6-77b39aa54533', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:58:22,089 [236] [DEBUG] [app] Ending request: urn:request:6ad98d6d-13cb-49a0-8d93-9659c177adbd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6ad98d6d-13cb-49a0-8d93-9659c177adbd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:22,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:22,090 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) notificationworker stdout | 2023-01-17 05:58:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:35.646354+00:00 (in 9.999576 seconds) notificationworker stdout | 2023-01-17 05:58:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:35 UTC)" (scheduled at 2023-01-17 05:58:25.646354+00:00) notificationworker stdout | 2023-01-17 05:58:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 25, 647145), True, datetime.datetime(2023, 1, 17, 5, 58, 25, 647145), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:35 UTC)" executed successfully builder stdout | 2023-01-17 05:58:26,255 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:26,268 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:26,268 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:26,278 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:58:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:58:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:35.853396+00:00 (in 6.889247 seconds) exportactionlogsworker stdout | 2023-01-17 05:58:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:28 UTC)" (scheduled at 2023-01-17 05:58:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:58:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:58:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:58:30,193 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:58:30,313 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:58:31,183 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:58:31,437 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:58:31,440 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:58:31,442 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:58:31,445 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:58:31,448 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:58:31,702 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:58:31,739 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:58:31,800 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:58:32,273 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 05:58:35,421 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:58:35,427 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} buildlogsarchiver stdout | 2023-01-17 05:58:35,429 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-secscan stdout | 2023-01-17 05:58:35,432 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} expiredappspecifictokenworker stdout | 2023-01-17 05:58:35,452 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:58:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:36.635997+00:00 (in 0.989183 seconds) notificationworker stdout | 2023-01-17 05:58:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:45 UTC)" (scheduled at 2023-01-17 05:58:35.646354+00:00) notificationworker stdout | 2023-01-17 05:58:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 35, 647221), True, datetime.datetime(2023, 1, 17, 5, 58, 35, 647221), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:58:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:58:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:28.963662+00:00 (in 53.109759 seconds) exportactionlogsworker stdout | 2023-01-17 05:58:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:35 UTC)" (scheduled at 2023-01-17 05:58:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:58:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:58:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 35, 854303), True, datetime.datetime(2023, 1, 17, 5, 58, 35, 854303), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:58:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:58:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:58:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:58:35,871 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:58:35,873 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:58:35,876 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:58:35,879 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:58:35,882 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:58:35,887 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:58:35,890 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:58:35,892 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:58:35,895 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 05:58:36,280 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:36,293 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:36,293 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:36,303 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:58:36,572 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:58:36,607 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} notificationworker stdout | 2023-01-17 05:58:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:45.646354+00:00 (in 9.009878 seconds) notificationworker stdout | 2023-01-17 05:58:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:03:36 UTC)" (scheduled at 2023-01-17 05:58:36.635997+00:00) notificationworker stdout | 2023-01-17 05:58:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 5, 58, 36, 636849), 'notification/%']) notificationworker stdout | 2023-01-17 05:58:36,647 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 58, 36, 636849), True, datetime.datetime(2023, 1, 17, 5, 58, 36, 636849), 0, 'notification/%']) notificationworker stdout | 2023-01-17 05:58:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 5, 58, 36, 636849), True, datetime.datetime(2023, 1, 17, 5, 58, 36, 636849), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 5, 58, 36, 636849), 'notification/%']) notificationworker stdout | 2023-01-17 05:58:36,654 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:03:36 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:58:36,943 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:58:37,060 [236] [DEBUG] [app] Starting request: urn:request:9660988a-c8b3-4a92-8e32-b8572c47efb1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:37,060 [235] [DEBUG] [app] Starting request: urn:request:016aebb2-72f5-4537-9290-bb54a7511ed9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:37,065 [230] [DEBUG] [app] Starting request: urn:request:b93797fc-15d9-4aff-8c88-274cf73a473e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:37,065 [230] [DEBUG] [app] Ending request: urn:request:b93797fc-15d9-4aff-8c88-274cf73a473e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b93797fc-15d9-4aff-8c88-274cf73a473e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:37,065 [242] [DEBUG] [app] Starting request: urn:request:30e70b09-652d-4acb-9ba2-83f55b578508 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:58:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:58:37,066 [242] [DEBUG] [app] Ending request: urn:request:30e70b09-652d-4acb-9ba2-83f55b578508 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30e70b09-652d-4acb-9ba2-83f55b578508', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:58:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:37,070 [235] [DEBUG] [app] Starting request: urn:request:f04ce461-501a-443f-889e-c0d10ce6ccdc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:37,071 [235] [DEBUG] [app] Ending request: urn:request:f04ce461-501a-443f-889e-c0d10ce6ccdc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f04ce461-501a-443f-889e-c0d10ce6ccdc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:37,071 [233] [DEBUG] [app] Starting request: urn:request:02a2ba2f-4f48-46ea-a258-c82975268032 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:37,072 [233] [DEBUG] [app] Ending request: urn:request:02a2ba2f-4f48-46ea-a258-c82975268032 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02a2ba2f-4f48-46ea-a258-c82975268032', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:58:37,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:37,087 [235] [DEBUG] [app] Ending request: urn:request:016aebb2-72f5-4537-9290-bb54a7511ed9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:016aebb2-72f5-4537-9290-bb54a7511ed9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:37,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:58:37,088 [236] [DEBUG] [app] Ending request: urn:request:9660988a-c8b3-4a92-8e32-b8572c47efb1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9660988a-c8b3-4a92-8e32-b8572c47efb1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 05:58:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:58:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:58:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:37.537407+00:00 (in 0.280347 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:58:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:37 UTC)" (scheduled at 2023-01-17 05:58:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:58:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:58:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 37, 257431), True, datetime.datetime(2023, 1, 17, 5, 58, 37, 257431), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:58:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:58:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:58:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:58:37,485 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} securityscanningnotificationworker stdout | 2023-01-17 05:58:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:58:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:37.256652+00:00 (in 59.718862 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:58:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:37 UTC)" (scheduled at 2023-01-17 05:58:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:58:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:58:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:37 UTC)" executed successfully manifestbackfillworker stdout | 2023-01-17 05:58:37,606 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:58:37,821 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:58:37,897 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:58:38,519 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:58:39,038 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:58:39,099 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:58:39,382 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:58:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:58:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:10.738568+00:00 (in 29.999581 seconds) buildlogsarchiver stdout | 2023-01-17 05:58:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:10 UTC)" (scheduled at 2023-01-17 05:58:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:58:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 58, 40, 739329), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:58:40,754 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:58:40,754 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:58:40,754 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:58:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:58:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:42.941536+00:00 (in 1.792825 seconds) namespacegcworker stdout | 2023-01-17 05:58:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:41 UTC)" (scheduled at 2023-01-17 05:58:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:58:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:58:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 41, 149116), True, datetime.datetime(2023, 1, 17, 5, 58, 41, 149116), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:58:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:58:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:58:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:58:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:58:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:41.148245+00:00 (in 58.206259 seconds) namespacegcworker stdout | 2023-01-17 05:58:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:42 UTC)" (scheduled at 2023-01-17 05:58:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:58:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:58:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 05:59:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:58:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:55.646354+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 05:58:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:55 UTC)" (scheduled at 2023-01-17 05:58:45.646354+00:00) notificationworker stdout | 2023-01-17 05:58:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 45, 647215), True, datetime.datetime(2023, 1, 17, 5, 58, 45, 647215), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:58:55 UTC)" executed successfully builder stdout | 2023-01-17 05:58:46,313 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:46,326 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:46,326 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:46,336 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:58:49.537472+00:00 (in 2.600018 seconds) securityworker stdout | 2023-01-17 05:58:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:16 UTC)" (scheduled at 2023-01-17 05:58:46.936988+00:00) securityworker stdout | 2023-01-17 05:58:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:58:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:58:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:16.936988+00:00 (in 27.399100 seconds) securityworker stdout | 2023-01-17 05:58:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:19 UTC)" (scheduled at 2023-01-17 05:58:49.537472+00:00) securityworker stdout | 2023-01-17 05:58:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:58:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:58:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:58:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:58:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:58:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:58:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:58:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:58:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:58:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:58:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:58:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:58:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:58:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:58:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:58:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:21.348605+00:00 (in 29.999490 seconds) gcworker stdout | 2023-01-17 05:58:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:21 UTC)" (scheduled at 2023-01-17 05:58:51.348605+00:00) gcworker stdout | 2023-01-17 05:58:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725531349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:58:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:58:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:58:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:58:52,060 [231] [DEBUG] [app] Starting request: urn:request:8de3841f-e7ef-4bb4-904c-7f6a597a8986 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:52,060 [236] [DEBUG] [app] Starting request: urn:request:c6d25d18-7675-4fca-ab6f-2155b9270fb4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:58:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:58:52,065 [230] [DEBUG] [app] Starting request: urn:request:4d4ba6bb-86c3-479b-822d-22de67e4630d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:52,065 [242] [DEBUG] [app] Starting request: urn:request:fd00bdfd-8f0e-49b1-8d4f-7617a59a2879 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:58:52,065 [230] [DEBUG] [app] Ending request: urn:request:4d4ba6bb-86c3-479b-822d-22de67e4630d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d4ba6bb-86c3-479b-822d-22de67e4630d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:58:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 05:58:52,065 [242] [DEBUG] [app] Ending request: urn:request:fd00bdfd-8f0e-49b1-8d4f-7617a59a2879 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd00bdfd-8f0e-49b1-8d4f-7617a59a2879', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:58:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:58:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:58:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:58:52,070 [231] [DEBUG] [app] Starting request: urn:request:4ed726ca-a88a-42bb-947f-13fb08576318 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:52,071 [231] [DEBUG] [app] Ending request: urn:request:4ed726ca-a88a-42bb-947f-13fb08576318 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4ed726ca-a88a-42bb-947f-13fb08576318', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:52,071 [235] [DEBUG] [app] Starting request: urn:request:caec58ab-e0fb-482a-99c3-f88ac23c909e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:58:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:58:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:58:52,071 [235] [DEBUG] [app] Ending request: urn:request:caec58ab-e0fb-482a-99c3-f88ac23c909e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:caec58ab-e0fb-482a-99c3-f88ac23c909e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:58:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:58:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:58:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:58:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:58:52,088 [236] [DEBUG] [app] Ending request: urn:request:c6d25d18-7675-4fca-ab6f-2155b9270fb4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c6d25d18-7675-4fca-ab6f-2155b9270fb4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:58:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:58:52,097 [231] [DEBUG] [app] Ending request: urn:request:8de3841f-e7ef-4bb4-904c-7f6a597a8986 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8de3841f-e7ef-4bb4-904c-7f6a597a8986', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:58:52,097 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.038 47 0.039) gunicorn-web stdout | 2023-01-17 05:58:52,097 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:58:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:58:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:58:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:01.636362+00:00 (in 5.989516 seconds) notificationworker stdout | 2023-01-17 05:58:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:05 UTC)" (scheduled at 2023-01-17 05:58:55.646354+00:00) notificationworker stdout | 2023-01-17 05:58:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:58:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 58, 55, 647255), True, datetime.datetime(2023, 1, 17, 5, 58, 55, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:58:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:58:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:58:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:05 UTC)" executed successfully builder stdout | 2023-01-17 05:58:56,338 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:58:56,351 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:58:56,351 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:58:56,361 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:00,229 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:59:00,320 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:59:01,218 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:59:01,444 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:59:01,447 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:59:01,450 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:59:01,452 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:59:01,454 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 05:59:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:05.646354+00:00 (in 4.009551 seconds) notificationworker stdout | 2023-01-17 05:59:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:01 UTC)" (scheduled at 2023-01-17 05:59:01.636362+00:00) notificationworker stdout | 2023-01-17 05:59:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 05:59:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:01 UTC)" executed successfully builder stdout | 2023-01-17 05:59:01,723 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:59:01,753 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:59:01,835 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:59:02,309 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 05:59:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:59:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:19.346120+00:00 (in 14.408839 seconds) repositorygcworker stdout | 2023-01-17 05:59:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:04 UTC)" (scheduled at 2023-01-17 05:59:04.936835+00:00) repositorygcworker stdout | 2023-01-17 05:59:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 05:59:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 4, 937738), True, datetime.datetime(2023, 1, 17, 5, 59, 4, 937738), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 05:59:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 05:59:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 05:59:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 05:59:05,428 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:59:05,433 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:59:05,437 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 05:59:05,459 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:59:05,486 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:59:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:15.646354+00:00 (in 9.999541 seconds) notificationworker stdout | 2023-01-17 05:59:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:15 UTC)" (scheduled at 2023-01-17 05:59:05.646354+00:00) notificationworker stdout | 2023-01-17 05:59:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 5, 647171), True, datetime.datetime(2023, 1, 17, 5, 59, 5, 647171), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:59:05,881 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:59:05,884 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:59:05,887 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:59:05,889 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:59:05,891 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:59:05,895 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:59:05,897 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:59:05,900 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:59:05,903 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 05:59:06,362 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:06,374 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:06,374 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:06,385 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:59:06,608 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:59:06,638 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:59:06,948 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:59:07,060 [231] [DEBUG] [app] Starting request: urn:request:51ba2e0d-1114-45f4-bb67-4af71db640be (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:07,060 [236] [DEBUG] [app] Starting request: urn:request:769eeb67-1d53-44f0-a596-4754c8e9d1c8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:07,064 [230] [DEBUG] [app] Starting request: urn:request:cb2c87c7-42b4-40d3-a1e4-be016a59a7fd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:07,064 [243] [DEBUG] [app] Starting request: urn:request:585a2eea-9377-44e7-9660-4a4a91765ab7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:07,065 [230] [DEBUG] [app] Ending request: urn:request:cb2c87c7-42b4-40d3-a1e4-be016a59a7fd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb2c87c7-42b4-40d3-a1e4-be016a59a7fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:59:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:59:07,065 [243] [DEBUG] [app] Ending request: urn:request:585a2eea-9377-44e7-9660-4a4a91765ab7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:585a2eea-9377-44e7-9660-4a4a91765ab7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:59:07,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:07,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:07,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:07,069 [235] [DEBUG] [app] Starting request: urn:request:1d5898de-d916-4268-a006-ffb256786496 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:07,069 [233] [DEBUG] [app] Starting request: urn:request:a6080c9f-c02d-4442-a1ff-077c12461ff8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:07,070 [235] [DEBUG] [app] Ending request: urn:request:1d5898de-d916-4268-a006-ffb256786496 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d5898de-d916-4268-a006-ffb256786496', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:07,070 [233] [DEBUG] [app] Ending request: urn:request:a6080c9f-c02d-4442-a1ff-077c12461ff8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6080c9f-c02d-4442-a1ff-077c12461ff8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 05:59:07,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:07,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:07,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:07,086 [231] [DEBUG] [app] Ending request: urn:request:51ba2e0d-1114-45f4-bb67-4af71db640be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:51ba2e0d-1114-45f4-bb67-4af71db640be', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:07,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:07,086 [236] [DEBUG] [app] Ending request: urn:request:769eeb67-1d53-44f0-a596-4754c8e9d1c8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:769eeb67-1d53-44f0-a596-4754c8e9d1c8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:07,086 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 05:59:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:59:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 05:59:07,521 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:59:07,639 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:59:07,839 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:59:07,932 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:59:08,549 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:59:09,063 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:59:09,134 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:59:09,417 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:59:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:59:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:40.738568+00:00 (in 29.999582 seconds) buildlogsarchiver stdout | 2023-01-17 05:59:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:40 UTC)" (scheduled at 2023-01-17 05:59:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:59:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 59, 10, 739434), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:59:10,758 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:59:10,758 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:59:10,758 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:25.646354+00:00 (in 9.999589 seconds) notificationworker stdout | 2023-01-17 05:59:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:25 UTC)" (scheduled at 2023-01-17 05:59:15.646354+00:00) notificationworker stdout | 2023-01-17 05:59:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 15, 647136), True, datetime.datetime(2023, 1, 17, 5, 59, 15, 647136), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:25 UTC)" executed successfully builder stdout | 2023-01-17 05:59:16,390 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:16,402 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:16,402 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:16,413 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:19.537472+00:00 (in 2.600053 seconds) securityworker stdout | 2023-01-17 05:59:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:46 UTC)" (scheduled at 2023-01-17 05:59:16.936988+00:00) securityworker stdout | 2023-01-17 05:59:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 05:59:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 05:59:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:04.936835+00:00 (in 45.590247 seconds) repositorygcworker stdout | 2023-01-17 05:59:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:19 UTC)" (scheduled at 2023-01-17 05:59:19.346120+00:00) repositorygcworker stdout | 2023-01-17 05:59:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 05:59:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:19 UTC)" executed successfully securityworker stdout | 2023-01-17 05:59:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:46.936988+00:00 (in 27.399053 seconds) securityworker stdout | 2023-01-17 05:59:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:49 UTC)" (scheduled at 2023-01-17 05:59:19.537472+00:00) securityworker stdout | 2023-01-17 05:59:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:59:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:59:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:59:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:59:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:59:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:59:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:49 UTC)" executed successfully gcworker stdout | 2023-01-17 05:59:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:59:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:51.348605+00:00 (in 29.999502 seconds) gcworker stdout | 2023-01-17 05:59:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:51 UTC)" (scheduled at 2023-01-17 05:59:21.348605+00:00) gcworker stdout | 2023-01-17 05:59:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725561349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:59:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:59:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:59:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 05:59:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:59:22,060 [231] [DEBUG] [app] Starting request: urn:request:fd7a7783-46a1-438c-b006-d2b5162db3ca (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:22,060 [235] [DEBUG] [app] Starting request: urn:request:b46c4499-b9e8-4e4c-a12f-2e1a574e5818 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:22,065 [230] [DEBUG] [app] Starting request: urn:request:cbf9b887-f536-4035-937a-56d1ce2313e0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:22,066 [243] [DEBUG] [app] Starting request: urn:request:26804e8a-c5e1-42ae-b81e-2de7db15dbaf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:22,066 [230] [DEBUG] [app] Ending request: urn:request:cbf9b887-f536-4035-937a-56d1ce2313e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cbf9b887-f536-4035-937a-56d1ce2313e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 05:59:22,066 [243] [DEBUG] [app] Ending request: urn:request:26804e8a-c5e1-42ae-b81e-2de7db15dbaf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:26804e8a-c5e1-42ae-b81e-2de7db15dbaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:59:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 05:59:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:22,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:22,071 [235] [DEBUG] [app] Starting request: urn:request:1dacf808-618c-41b4-9608-28479c07917c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:22,071 [235] [DEBUG] [app] Ending request: urn:request:1dacf808-618c-41b4-9608-28479c07917c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1dacf808-618c-41b4-9608-28479c07917c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:22,072 [233] [DEBUG] [app] Starting request: urn:request:480fc977-db5b-440e-93b2-a98cddd74079 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:22,072 [233] [DEBUG] [app] Ending request: urn:request:480fc977-db5b-440e-93b2-a98cddd74079 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:480fc977-db5b-440e-93b2-a98cddd74079', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:22,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:22,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:22,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:22,089 [231] [DEBUG] [app] Ending request: urn:request:fd7a7783-46a1-438c-b006-d2b5162db3ca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd7a7783-46a1-438c-b006-d2b5162db3ca', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:22,089 [235] [DEBUG] [app] Ending request: urn:request:b46c4499-b9e8-4e4c-a12f-2e1a574e5818 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b46c4499-b9e8-4e4c-a12f-2e1a574e5818', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 05:59:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) notificationworker stdout | 2023-01-17 05:59:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:35.646354+00:00 (in 9.999573 seconds) notificationworker stdout | 2023-01-17 05:59:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:35 UTC)" (scheduled at 2023-01-17 05:59:25.646354+00:00) notificationworker stdout | 2023-01-17 05:59:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 25, 647141), True, datetime.datetime(2023, 1, 17, 5, 59, 25, 647141), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:35 UTC)" executed successfully builder stdout | 2023-01-17 05:59:26,422 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:26,435 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:26,435 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:26,445 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:59:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:59:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:35.853396+00:00 (in 6.889252 seconds) exportactionlogsworker stdout | 2023-01-17 05:59:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:28 UTC)" (scheduled at 2023-01-17 05:59:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 05:59:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 05:59:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:30,254 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 05:59:30,355 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 05:59:31,252 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 05:59:31,451 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 05:59:31,453 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 05:59:31,456 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 05:59:31,458 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 05:59:31,461 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 05:59:31,733 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 05:59:31,770 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 05:59:31,855 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 05:59:32,334 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 05:59:35,434 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 05:59:35,438 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 05:59:35,443 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 05:59:35,478 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 05:59:35,522 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 05:59:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:45.646354+00:00 (in 9.999589 seconds) notificationworker stdout | 2023-01-17 05:59:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:45 UTC)" (scheduled at 2023-01-17 05:59:35.646354+00:00) notificationworker stdout | 2023-01-17 05:59:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 35, 647129), True, datetime.datetime(2023, 1, 17, 5, 59, 35, 647129), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 05:59:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 05:59:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:28.963662+00:00 (in 53.109741 seconds) exportactionlogsworker stdout | 2023-01-17 05:59:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:35 UTC)" (scheduled at 2023-01-17 05:59:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 05:59:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 05:59:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 35, 854342), True, datetime.datetime(2023, 1, 17, 5, 59, 35, 854342), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 05:59:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 05:59:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 05:59:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 05:59:35,888 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 05:59:35,892 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 05:59:35,894 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 05:59:35,898 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 05:59:35,901 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 05:59:35,904 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 05:59:35,906 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 05:59:35,909 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 05:59:35,911 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 05:59:36,455 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:36,467 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:36,468 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:36,478 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 05:59:36,622 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 05:59:36,655 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 05:59:36,983 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 05:59:37,060 [231] [DEBUG] [app] Starting request: urn:request:bd523912-631e-4b46-a98e-e2534581fd7f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:37,060 [236] [DEBUG] [app] Starting request: urn:request:a6321c3f-b7d1-4833-93f1-8571a6a3217d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:37,065 [230] [DEBUG] [app] Starting request: urn:request:653f9a8c-03af-481c-8ca3-1ba2e39fa0a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:37,066 [241] [DEBUG] [app] Starting request: urn:request:34eb1fd0-49ad-4594-a566-48532b5e40d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:37,066 [230] [DEBUG] [app] Ending request: urn:request:653f9a8c-03af-481c-8ca3-1ba2e39fa0a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:653f9a8c-03af-481c-8ca3-1ba2e39fa0a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 05:59:37,066 [241] [DEBUG] [app] Ending request: urn:request:34eb1fd0-49ad-4594-a566-48532b5e40d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34eb1fd0-49ad-4594-a566-48532b5e40d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:37,067 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 05:59:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:37,071 [235] [DEBUG] [app] Starting request: urn:request:00c3987e-882b-4e1d-b935-c2e9fcfee7b2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:37,071 [231] [DEBUG] [app] Starting request: urn:request:d67bfb30-6a8f-4d3a-8303-07a05011732a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:37,071 [235] [DEBUG] [app] Ending request: urn:request:00c3987e-882b-4e1d-b935-c2e9fcfee7b2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00c3987e-882b-4e1d-b935-c2e9fcfee7b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:37,071 [231] [DEBUG] [app] Ending request: urn:request:d67bfb30-6a8f-4d3a-8303-07a05011732a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d67bfb30-6a8f-4d3a-8303-07a05011732a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 05:59:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:37,088 [231] [DEBUG] [app] Ending request: urn:request:bd523912-631e-4b46-a98e-e2534581fd7f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd523912-631e-4b46-a98e-e2534581fd7f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:37,088 [236] [DEBUG] [app] Ending request: urn:request:a6321c3f-b7d1-4833-93f1-8571a6a3217d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6321c3f-b7d1-4833-93f1-8571a6a3217d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 05:59:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 05:59:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:59:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:37.537407+00:00 (in 0.280359 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:59:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:37 UTC)" (scheduled at 2023-01-17 05:59:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:59:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 05:59:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 37, 257416), True, datetime.datetime(2023, 1, 17, 5, 59, 37, 257416), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 05:59:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 05:59:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 05:59:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 05:59:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 05:59:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:37.256652+00:00 (in 59.718911 seconds) securityscanningnotificationworker stdout | 2023-01-17 05:59:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:37 UTC)" (scheduled at 2023-01-17 05:59:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 05:59:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 05:59:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 05:59:37,543 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 05:59:37,673 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 05:59:37,869 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 05:59:37,967 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 05:59:38,574 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 05:59:39,091 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 05:59:39,149 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 05:59:39,424 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 05:59:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 05:59:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:10.738568+00:00 (in 29.999494 seconds) buildlogsarchiver stdout | 2023-01-17 05:59:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:10 UTC)" (scheduled at 2023-01-17 05:59:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 05:59:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 5, 59, 40, 739543), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 05:59:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 05:59:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 05:59:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:59:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:59:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:42.941536+00:00 (in 1.792830 seconds) namespacegcworker stdout | 2023-01-17 05:59:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:41 UTC)" (scheduled at 2023-01-17 05:59:41.148245+00:00) namespacegcworker stdout | 2023-01-17 05:59:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 05:59:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 41, 149101), True, datetime.datetime(2023, 1, 17, 5, 59, 41, 149101), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 05:59:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 05:59:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 05:59:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 05:59:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 05:59:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:41.148245+00:00 (in 58.206229 seconds) namespacegcworker stdout | 2023-01-17 05:59:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:42 UTC)" (scheduled at 2023-01-17 05:59:42.941536+00:00) namespacegcworker stdout | 2023-01-17 05:59:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 05:59:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:00:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 05:59:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:55.646354+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 05:59:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:55 UTC)" (scheduled at 2023-01-17 05:59:45.646354+00:00) notificationworker stdout | 2023-01-17 05:59:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 45, 647151), True, datetime.datetime(2023, 1, 17, 5, 59, 45, 647151), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 05:59:55 UTC)" executed successfully builder stdout | 2023-01-17 05:59:46,488 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:46,501 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:46,501 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:46,512 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 05:59:49.537472+00:00 (in 2.600016 seconds) securityworker stdout | 2023-01-17 05:59:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:16 UTC)" (scheduled at 2023-01-17 05:59:46.936988+00:00) securityworker stdout | 2023-01-17 05:59:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:16 UTC)" executed successfully securityworker stdout | 2023-01-17 05:59:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 05:59:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:16.936988+00:00 (in 27.399081 seconds) securityworker stdout | 2023-01-17 05:59:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:19 UTC)" (scheduled at 2023-01-17 05:59:49.537472+00:00) securityworker stdout | 2023-01-17 05:59:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 05:59:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 05:59:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:49,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 05:59:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 05:59:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 05:59:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 05:59:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 05:59:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 05:59:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 05:59:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 05:59:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 05:59:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 05:59:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 05:59:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:19 UTC)" executed successfully gcworker stdout | 2023-01-17 05:59:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 05:59:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:21.348605+00:00 (in 29.999521 seconds) gcworker stdout | 2023-01-17 05:59:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:21 UTC)" (scheduled at 2023-01-17 05:59:51.348605+00:00) gcworker stdout | 2023-01-17 05:59:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725591349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 05:59:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 05:59:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 05:59:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 05:59:52,059 [231] [DEBUG] [app] Starting request: urn:request:2b6281ab-c023-4a79-8f2d-834f7c208c7d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:52,059 [236] [DEBUG] [app] Starting request: urn:request:97819a4e-795a-44ed-a495-df4f038ab5d1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 05:59:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 05:59:52,065 [230] [DEBUG] [app] Starting request: urn:request:dac94847-daaf-4ce1-9611-533c3a5e5a24 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:52,065 [230] [DEBUG] [app] Ending request: urn:request:dac94847-daaf-4ce1-9611-533c3a5e5a24 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dac94847-daaf-4ce1-9611-533c3a5e5a24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:52,065 [239] [DEBUG] [app] Starting request: urn:request:c424ad07-5c08-4967-a2fa-f8924e93c248 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 05:59:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 05:59:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 05:59:52,066 [239] [DEBUG] [app] Ending request: urn:request:c424ad07-5c08-4967-a2fa-f8924e93c248 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c424ad07-5c08-4967-a2fa-f8924e93c248', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 05:59:52,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 05:59:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 05:59:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 05:59:52,070 [236] [DEBUG] [app] Starting request: urn:request:57846c4b-139f-4682-b195-0074863fe775 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:52,070 [235] [DEBUG] [app] Starting request: urn:request:4c6d73b6-cb6e-4e5c-86c9-6c6b99f59b3a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 05:59:52,070 [236] [DEBUG] [app] Ending request: urn:request:57846c4b-139f-4682-b195-0074863fe775 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57846c4b-139f-4682-b195-0074863fe775', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:52,071 [235] [DEBUG] [app] Ending request: urn:request:4c6d73b6-cb6e-4e5c-86c9-6c6b99f59b3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c6d73b6-cb6e-4e5c-86c9-6c6b99f59b3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 05:59:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 05:59:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:05:59:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 05:59:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 05:59:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 05:59:52,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:52,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 05:59:52,089 [236] [DEBUG] [app] Ending request: urn:request:97819a4e-795a-44ed-a495-df4f038ab5d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97819a4e-795a-44ed-a495-df4f038ab5d1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:52,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:52,089 [231] [DEBUG] [app] Ending request: urn:request:2b6281ab-c023-4a79-8f2d-834f7c208c7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b6281ab-c023-4a79-8f2d-834f7c208c7d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 05:59:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 05:59:52,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.031 47 0.031) gunicorn-web stdout | 2023-01-17 05:59:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:05:59:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 05:59:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 05:59:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:01.636362+00:00 (in 5.989580 seconds) notificationworker stdout | 2023-01-17 05:59:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:05 UTC)" (scheduled at 2023-01-17 05:59:55.646354+00:00) notificationworker stdout | 2023-01-17 05:59:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 05:59:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 5, 59, 55, 647141), True, datetime.datetime(2023, 1, 17, 5, 59, 55, 647141), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 05:59:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 05:59:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 05:59:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:05 UTC)" executed successfully builder stdout | 2023-01-17 05:59:56,522 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 05:59:56,534 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 05:59:56,534 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 05:59:56,545 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:00,290 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:00:00,388 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 06:00:01,286 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 06:00:01,456 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:00:01,460 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:00:01,462 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:00:01,465 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:00:01,468 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:00:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:05.646354+00:00 (in 4.009537 seconds) notificationworker stdout | 2023-01-17 06:00:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:01 UTC)" (scheduled at 2023-01-17 06:00:01.636362+00:00) notificationworker stdout | 2023-01-17 06:00:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:00:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:01 UTC)" executed successfully builder stdout | 2023-01-17 06:00:01,769 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 06:00:01,791 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:00:01,890 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:00:02,369 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:00:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:00:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:19.346120+00:00 (in 14.408861 seconds) repositorygcworker stdout | 2023-01-17 06:00:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:04 UTC)" (scheduled at 2023-01-17 06:00:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:00:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:00:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 4, 937671), True, datetime.datetime(2023, 1, 17, 6, 0, 4, 937671), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:00:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:00:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:00:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:00:05,441 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:00:05,444 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:00:05,448 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 06:00:05,485 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:00:05,549 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 06:00:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:15.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 06:00:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:15 UTC)" (scheduled at 2023-01-17 06:00:05.646354+00:00) notificationworker stdout | 2023-01-17 06:00:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 5, 647327), True, datetime.datetime(2023, 1, 17, 6, 0, 5, 647327), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:00:05,894 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:00:05,899 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:00:05,901 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:00:05,904 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:00:05,910 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:00:05,912 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:00:05,914 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:00:05,917 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:00:05,919 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 06:00:06,555 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:06,568 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:06,568 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:06,578 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 06:00:06,652 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:00:06,683 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:00:07,019 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 06:00:07,060 [235] [DEBUG] [app] Starting request: urn:request:bed46469-1401-4e4c-8478-272b7e1e2d42 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:07,060 [231] [DEBUG] [app] Starting request: urn:request:9db3f103-2c88-4541-a99a-4ca984ddec0c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:07,064 [230] [DEBUG] [app] Starting request: urn:request:7cddd984-0f54-4414-b943-f75b19159b8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:07,065 [242] [DEBUG] [app] Starting request: urn:request:93289d51-303c-4c1e-afe4-627a91fd9f59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:07,065 [230] [DEBUG] [app] Ending request: urn:request:7cddd984-0f54-4414-b943-f75b19159b8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7cddd984-0f54-4414-b943-f75b19159b8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:00:07,065 [242] [DEBUG] [app] Ending request: urn:request:93289d51-303c-4c1e-afe4-627a91fd9f59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93289d51-303c-4c1e-afe4-627a91fd9f59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:00:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:07,070 [236] [DEBUG] [app] Starting request: urn:request:97ccce59-1df4-48de-8155-a997c3e65f51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:07,070 [233] [DEBUG] [app] Starting request: urn:request:fcb5b1a0-d9ad-4d90-bb92-98b389497839 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:07,070 [236] [DEBUG] [app] Ending request: urn:request:97ccce59-1df4-48de-8155-a997c3e65f51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97ccce59-1df4-48de-8155-a997c3e65f51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:07,070 [233] [DEBUG] [app] Ending request: urn:request:fcb5b1a0-d9ad-4d90-bb92-98b389497839 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fcb5b1a0-d9ad-4d90-bb92-98b389497839', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:07,087 [231] [DEBUG] [app] Ending request: urn:request:9db3f103-2c88-4541-a99a-4ca984ddec0c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9db3f103-2c88-4541-a99a-4ca984ddec0c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:07,087 [235] [DEBUG] [app] Ending request: urn:request:bed46469-1401-4e4c-8478-272b7e1e2d42 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bed46469-1401-4e4c-8478-272b7e1e2d42', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:00:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" chunkcleanupworker stdout | 2023-01-17 06:00:07,557 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:00:07,708 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:00:07,894 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:00:07,997 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:00:08,601 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:00:09,105 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:00:09,177 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:00:09,459 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:00:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:00:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:40.738568+00:00 (in 29.999428 seconds) buildlogsarchiver stdout | 2023-01-17 06:00:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:40 UTC)" (scheduled at 2023-01-17 06:00:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:00:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 0, 10, 739622), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:00:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:00:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:00:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:00:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:25.646354+00:00 (in 9.999565 seconds) notificationworker stdout | 2023-01-17 06:00:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:25 UTC)" (scheduled at 2023-01-17 06:00:15.646354+00:00) notificationworker stdout | 2023-01-17 06:00:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 15, 647187), True, datetime.datetime(2023, 1, 17, 6, 0, 15, 647187), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:25 UTC)" executed successfully builder stdout | 2023-01-17 06:00:16,589 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:16,601 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:16,601 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:16,612 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:19.537472+00:00 (in 2.600069 seconds) securityworker stdout | 2023-01-17 06:00:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:46 UTC)" (scheduled at 2023-01-17 06:00:16.936988+00:00) securityworker stdout | 2023-01-17 06:00:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:00:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:00:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:04.936835+00:00 (in 45.590275 seconds) repositorygcworker stdout | 2023-01-17 06:00:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:19 UTC)" (scheduled at 2023-01-17 06:00:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:00:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:00:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:00:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:46.936988+00:00 (in 27.399095 seconds) securityworker stdout | 2023-01-17 06:00:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:49 UTC)" (scheduled at 2023-01-17 06:00:19.537472+00:00) securityworker stdout | 2023-01-17 06:00:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:00:19,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:00:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:00:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:00:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:00:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:00:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:00:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:00:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:51.348605+00:00 (in 29.999521 seconds) gcworker stdout | 2023-01-17 06:00:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:51 UTC)" (scheduled at 2023-01-17 06:00:21.348605+00:00) gcworker stdout | 2023-01-17 06:00:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725621349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:00:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:00:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:00:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:00:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:00:22,059 [233] [DEBUG] [app] Starting request: urn:request:b2509cc2-cef2-438c-a15a-7aefe641bc9a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:22,060 [231] [DEBUG] [app] Starting request: urn:request:69654306-c2fb-40c3-bac4-22183b2b429d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:22,064 [243] [DEBUG] [app] Starting request: urn:request:fdcdb304-d3ea-4853-b453-e1ac0494c9cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:22,065 [230] [DEBUG] [app] Starting request: urn:request:98915ea3-4125-46ce-aaf1-1ef58370f480 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:22,065 [243] [DEBUG] [app] Ending request: urn:request:fdcdb304-d3ea-4853-b453-e1ac0494c9cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fdcdb304-d3ea-4853-b453-e1ac0494c9cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:00:22,065 [230] [DEBUG] [app] Ending request: urn:request:98915ea3-4125-46ce-aaf1-1ef58370f480 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:98915ea3-4125-46ce-aaf1-1ef58370f480', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:00:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:00:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:22,070 [236] [DEBUG] [app] Starting request: urn:request:f75a1565-17fe-4269-b329-e8859cb1538f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:22,070 [231] [DEBUG] [app] Starting request: urn:request:72b4b2d6-564f-4134-aeca-56369e095428 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:22,070 [236] [DEBUG] [app] Ending request: urn:request:f75a1565-17fe-4269-b329-e8859cb1538f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f75a1565-17fe-4269-b329-e8859cb1538f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:22,070 [231] [DEBUG] [app] Ending request: urn:request:72b4b2d6-564f-4134-aeca-56369e095428 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:72b4b2d6-564f-4134-aeca-56369e095428', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:22,088 [231] [DEBUG] [app] Ending request: urn:request:69654306-c2fb-40c3-bac4-22183b2b429d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69654306-c2fb-40c3-bac4-22183b2b429d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:22,088 [233] [DEBUG] [app] Ending request: urn:request:b2509cc2-cef2-438c-a15a-7aefe641bc9a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2509cc2-cef2-438c-a15a-7aefe641bc9a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 06:00:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:00:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:35.646354+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 06:00:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:35 UTC)" (scheduled at 2023-01-17 06:00:25.646354+00:00) notificationworker stdout | 2023-01-17 06:00:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 25, 647204), True, datetime.datetime(2023, 1, 17, 6, 0, 25, 647204), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:35 UTC)" executed successfully builder stdout | 2023-01-17 06:00:26,622 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:26,635 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:26,635 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:26,645 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:00:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:00:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:35.853396+00:00 (in 6.889284 seconds) exportactionlogsworker stdout | 2023-01-17 06:00:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:28 UTC)" (scheduled at 2023-01-17 06:00:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:00:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:00:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:00:30,326 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:00:30,401 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 06:00:31,322 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 06:00:31,463 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:00:31,465 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:00:31,469 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:00:31,471 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:00:31,474 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} builder stdout | 2023-01-17 06:00:31,801 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} logrotateworker stdout | 2023-01-17 06:00:31,825 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:00:31,911 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:00:32,397 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:00:35,449 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:00:35,451 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:00:35,454 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 06:00:35,520 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:00:35,582 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 06:00:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:45.646354+00:00 (in 9.999579 seconds) notificationworker stdout | 2023-01-17 06:00:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:45 UTC)" (scheduled at 2023-01-17 06:00:35.646354+00:00) notificationworker stdout | 2023-01-17 06:00:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 35, 647184), True, datetime.datetime(2023, 1, 17, 6, 0, 35, 647184), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:00:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:00:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:28.963662+00:00 (in 53.109793 seconds) exportactionlogsworker stdout | 2023-01-17 06:00:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:35 UTC)" (scheduled at 2023-01-17 06:00:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:00:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:00:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 35, 854220), True, datetime.datetime(2023, 1, 17, 6, 0, 35, 854220), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:00:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:00:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:00:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:00:35,902 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:00:35,906 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:00:35,909 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:00:35,911 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:00:35,917 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:00:35,919 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:00:35,922 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:00:35,924 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:00:35,926 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 06:00:36,650 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:36,663 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:36,663 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:36,673 [60] [DEBUG] [data.database] Disconnecting from database. storagereplication stdout | 2023-01-17 06:00:36,685 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:00:36,717 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:00:37,054 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} gunicorn-web stdout | 2023-01-17 06:00:37,060 [231] [DEBUG] [app] Starting request: urn:request:84545f36-0f39-4125-96ba-7b640796bf47 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:37,060 [236] [DEBUG] [app] Starting request: urn:request:c2d8e18e-872d-4440-92f7-f112a9d8a3e4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:37,065 [230] [DEBUG] [app] Starting request: urn:request:5f9434d2-1e99-41e4-bfdd-3b2319fba580 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:37,065 [243] [DEBUG] [app] Starting request: urn:request:57864963-8198-4484-a869-805ae9cabf12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:37,065 [230] [DEBUG] [app] Ending request: urn:request:5f9434d2-1e99-41e4-bfdd-3b2319fba580 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f9434d2-1e99-41e4-bfdd-3b2319fba580', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:00:37,066 [243] [DEBUG] [app] Ending request: urn:request:57864963-8198-4484-a869-805ae9cabf12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57864963-8198-4484-a869-805ae9cabf12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:00:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:37,070 [233] [DEBUG] [app] Starting request: urn:request:f461b51f-3908-4e54-8779-b2421b0fd175 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:37,070 [231] [DEBUG] [app] Starting request: urn:request:bdd55063-8979-417b-8402-ac974d2fcaf8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:37,071 [233] [DEBUG] [app] Ending request: urn:request:f461b51f-3908-4e54-8779-b2421b0fd175 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f461b51f-3908-4e54-8779-b2421b0fd175', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:37,071 [231] [DEBUG] [app] Ending request: urn:request:bdd55063-8979-417b-8402-ac974d2fcaf8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bdd55063-8979-417b-8402-ac974d2fcaf8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:00:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:37,087 [236] [DEBUG] [app] Ending request: urn:request:c2d8e18e-872d-4440-92f7-f112a9d8a3e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2d8e18e-872d-4440-92f7-f112a9d8a3e4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:37,087 [231] [DEBUG] [app] Ending request: urn:request:84545f36-0f39-4125-96ba-7b640796bf47 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84545f36-0f39-4125-96ba-7b640796bf47', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:00:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) securityscanningnotificationworker stdout | 2023-01-17 06:00:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:00:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:37.537407+00:00 (in 0.280386 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:00:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:37 UTC)" (scheduled at 2023-01-17 06:00:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:00:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:00:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 37, 257390), True, datetime.datetime(2023, 1, 17, 6, 0, 37, 257390), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:00:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:00:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:00:37,270 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:00:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:00:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:23.537048+00:00 (in 45.999271 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:00:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:37 UTC)" (scheduled at 2023-01-17 06:00:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:00:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:00:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 06:00:37,589 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:00:37,742 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:00:37,928 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:00:38,027 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:00:38,607 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:00:39,127 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:00:39,205 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:00:39,465 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:00:40,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:00:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:10.738568+00:00 (in 29.999406 seconds) buildlogsarchiver stdout | 2023-01-17 06:00:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:10 UTC)" (scheduled at 2023-01-17 06:00:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:00:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 0, 40, 739647), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:00:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:00:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:00:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:00:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:00:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:42.941536+00:00 (in 1.792807 seconds) namespacegcworker stdout | 2023-01-17 06:00:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:41 UTC)" (scheduled at 2023-01-17 06:00:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:00:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:00:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 41, 149124), True, datetime.datetime(2023, 1, 17, 6, 0, 41, 149124), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:00:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:00:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:00:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:00:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:00:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:41.148245+00:00 (in 58.206263 seconds) namespacegcworker stdout | 2023-01-17 06:00:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:42 UTC)" (scheduled at 2023-01-17 06:00:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:00:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:00:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:01:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:00:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:55.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 06:00:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:55 UTC)" (scheduled at 2023-01-17 06:00:45.646354+00:00) notificationworker stdout | 2023-01-17 06:00:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 45, 647159), True, datetime.datetime(2023, 1, 17, 6, 0, 45, 647159), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:00:55 UTC)" executed successfully builder stdout | 2023-01-17 06:00:46,674 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:46,687 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:46,687 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:46,698 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:00:49.537472+00:00 (in 2.600045 seconds) securityworker stdout | 2023-01-17 06:00:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:16 UTC)" (scheduled at 2023-01-17 06:00:46.936988+00:00) securityworker stdout | 2023-01-17 06:00:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:00:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:00:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:16.936988+00:00 (in 27.399094 seconds) securityworker stdout | 2023-01-17 06:00:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:19 UTC)" (scheduled at 2023-01-17 06:00:49.537472+00:00) securityworker stdout | 2023-01-17 06:00:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:00:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:00:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:00:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:00:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:00:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:00:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:00:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:00:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:00:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:00:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:00:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:00:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:00:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:00:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:00:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:21.348605+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 06:00:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:21 UTC)" (scheduled at 2023-01-17 06:00:51.348605+00:00) gcworker stdout | 2023-01-17 06:00:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725651349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:00:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:00:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:00:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:00:52,060 [231] [DEBUG] [app] Starting request: urn:request:8b64421f-48c2-4418-982c-2e7f95f7f6f3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:52,060 [233] [DEBUG] [app] Starting request: urn:request:b850c20d-4540-45d1-a9e0-3c728fc288ab (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:00:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:00:52,065 [241] [DEBUG] [app] Starting request: urn:request:ee737808-7d49-4099-ae14-2f8d67ac96da (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:52,065 [230] [DEBUG] [app] Starting request: urn:request:c2ec2c0d-904d-4abc-8a11-9a5c5487a626 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:00:52,065 [241] [DEBUG] [app] Ending request: urn:request:ee737808-7d49-4099-ae14-2f8d67ac96da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee737808-7d49-4099-ae14-2f8d67ac96da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:52,066 [230] [DEBUG] [app] Ending request: urn:request:c2ec2c0d-904d-4abc-8a11-9a5c5487a626 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2ec2c0d-904d-4abc-8a11-9a5c5487a626', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:00:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:00:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.003) gunicorn-web stdout | 2023-01-17 06:00:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:00:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:00:52,070 [236] [DEBUG] [app] Starting request: urn:request:3bede04b-2d2c-4493-8258-b6c6c7934484 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:52,071 [233] [DEBUG] [app] Starting request: urn:request:e7fbd4d8-e9be-43e7-8d1d-cfa734f30f36 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:00:52,071 [233] [DEBUG] [app] Ending request: urn:request:e7fbd4d8-e9be-43e7-8d1d-cfa734f30f36 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7fbd4d8-e9be-43e7-8d1d-cfa734f30f36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:00:52,071 [236] [DEBUG] [app] Ending request: urn:request:3bede04b-2d2c-4493-8258-b6c6c7934484 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3bede04b-2d2c-4493-8258-b6c6c7934484', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:00:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:00:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:00:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:00:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:00:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:00:52,087 [233] [DEBUG] [app] Ending request: urn:request:b850c20d-4540-45d1-a9e0-3c728fc288ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b850c20d-4540-45d1-a9e0-3c728fc288ab', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:52,088 [231] [DEBUG] [app] Ending request: urn:request:8b64421f-48c2-4418-982c-2e7f95f7f6f3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b64421f-48c2-4418-982c-2e7f95f7f6f3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:00:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:00:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:00:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:00:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 06:00:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:00:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:01.636362+00:00 (in 5.989577 seconds) notificationworker stdout | 2023-01-17 06:00:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:05 UTC)" (scheduled at 2023-01-17 06:00:55.646354+00:00) notificationworker stdout | 2023-01-17 06:00:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:00:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 0, 55, 647199), True, datetime.datetime(2023, 1, 17, 6, 0, 55, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:00:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:00:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:00:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:05 UTC)" executed successfully builder stdout | 2023-01-17 06:00:56,708 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:00:56,720 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:00:56,721 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:00:56,731 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:00,332 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:01:00,406 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 06:01:01,358 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 06:01:01,470 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:01:01,472 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:01:01,475 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:01:01,477 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:01:01,480 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:01:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:05.646354+00:00 (in 4.009522 seconds) notificationworker stdout | 2023-01-17 06:01:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:01 UTC)" (scheduled at 2023-01-17 06:01:01.636362+00:00) notificationworker stdout | 2023-01-17 06:01:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:01:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:01 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:01:01,833 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:01:01,836 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:01:01,916 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:01:02,432 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:01:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:01:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:17.345601+00:00 (in 12.408359 seconds) repositorygcworker stdout | 2023-01-17 06:01:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:04 UTC)" (scheduled at 2023-01-17 06:01:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:01:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:01:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 4, 937657), True, datetime.datetime(2023, 1, 17, 6, 1, 4, 937657), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:01:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:01:04,948 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:01:04,948 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:01:05,455 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:01:05,458 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:01:05,460 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 06:01:05,548 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:01:05,616 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 06:01:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:15.646354+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 06:01:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:15 UTC)" (scheduled at 2023-01-17 06:01:05.646354+00:00) notificationworker stdout | 2023-01-17 06:01:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 5, 647199), True, datetime.datetime(2023, 1, 17, 6, 1, 5, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:01:05,913 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:01:05,916 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:01:05,918 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:01:05,920 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:01:05,924 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:01:05,926 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:01:05,928 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:01:05,931 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:01:05,933 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:01:06,698 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:01:06,731 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 06:01:06,736 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:06,750 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:06,750 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:06,761 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:07,060 [231] [DEBUG] [app] Starting request: urn:request:bb07fe8c-4d0a-4244-95dd-53abbf9d01b7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:07,060 [235] [DEBUG] [app] Starting request: urn:request:a2d1990f-a50d-469d-9663-e2e812923574 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:07,064 [241] [DEBUG] [app] Starting request: urn:request:c41ae1df-cbe1-45a1-b2de-dc657c282fc9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:07,065 [241] [DEBUG] [app] Ending request: urn:request:c41ae1df-cbe1-45a1-b2de-dc657c282fc9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c41ae1df-cbe1-45a1-b2de-dc657c282fc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:01:07,065 [243] [DEBUG] [app] Starting request: urn:request:2ce9166a-73e1-42a7-ab87-b5e156744211 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:07,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:01:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:01:07,066 [243] [DEBUG] [app] Ending request: urn:request:2ce9166a-73e1-42a7-ab87-b5e156744211 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ce9166a-73e1-42a7-ab87-b5e156744211', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:01:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:07,070 [236] [DEBUG] [app] Starting request: urn:request:64bab6b6-bf6c-41de-ac29-7604de321684 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:07,071 [231] [DEBUG] [app] Starting request: urn:request:ab51f158-575c-4127-8fdb-09f7e1008d31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:07,071 [236] [DEBUG] [app] Ending request: urn:request:64bab6b6-bf6c-41de-ac29-7604de321684 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64bab6b6-bf6c-41de-ac29-7604de321684', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:07,071 [231] [DEBUG] [app] Ending request: urn:request:ab51f158-575c-4127-8fdb-09f7e1008d31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ab51f158-575c-4127-8fdb-09f7e1008d31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:01:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:07,088 [231] [DEBUG] [app] Ending request: urn:request:bb07fe8c-4d0a-4244-95dd-53abbf9d01b7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb07fe8c-4d0a-4244-95dd-53abbf9d01b7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:01:07,088 [235] [DEBUG] [app] Ending request: urn:request:a2d1990f-a50d-469d-9663-e2e812923574 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2d1990f-a50d-469d-9663-e2e812923574', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:01:07,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:01:07,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" repositorygcworker stdout | 2023-01-17 06:01:07,090 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:01:07,620 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:01:07,776 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:01:07,939 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:01:08,063 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:01:08,643 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:01:09,134 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:01:09,239 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:01:09,500 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:01:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:01:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:40.738568+00:00 (in 29.999326 seconds) buildlogsarchiver stdout | 2023-01-17 06:01:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:40 UTC)" (scheduled at 2023-01-17 06:01:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:01:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 1, 10, 739723), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:01:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:01:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:01:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:25.646354+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 06:01:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:25 UTC)" (scheduled at 2023-01-17 06:01:15.646354+00:00) notificationworker stdout | 2023-01-17 06:01:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 15, 647213), True, datetime.datetime(2023, 1, 17, 6, 1, 15, 647213), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:25 UTC)" executed successfully builder stdout | 2023-01-17 06:01:16,771 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:16,784 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:16,784 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:16,795 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:19.537472+00:00 (in 2.600022 seconds) securityworker stdout | 2023-01-17 06:01:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:46 UTC)" (scheduled at 2023-01-17 06:01:16.936988+00:00) securityworker stdout | 2023-01-17 06:01:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:01:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:01:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:19.346120+00:00 (in 2.000046 seconds) repositorygcworker stdout | 2023-01-17 06:01:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:17 UTC)" (scheduled at 2023-01-17 06:01:17.345601+00:00) repositorygcworker stdout | 2023-01-17 06:01:17,346 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 1, 17, 346443), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:01:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 17, 346443), True, datetime.datetime(2023, 1, 17, 6, 1, 17, 346443), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:01:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 17, 346443), True, datetime.datetime(2023, 1, 17, 6, 1, 17, 346443), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 1, 17, 346443), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:01:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:01:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:01:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:01:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:04.936835+00:00 (in 45.590289 seconds) repositorygcworker stdout | 2023-01-17 06:01:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:19 UTC)" (scheduled at 2023-01-17 06:01:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:01:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:01:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:01:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:46.936988+00:00 (in 27.399076 seconds) securityworker stdout | 2023-01-17 06:01:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:49 UTC)" (scheduled at 2023-01-17 06:01:19.537472+00:00) securityworker stdout | 2023-01-17 06:01:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:01:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:01:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:01:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:01:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:01:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:01:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:01:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:01:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:51.348605+00:00 (in 29.999516 seconds) gcworker stdout | 2023-01-17 06:01:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:51 UTC)" (scheduled at 2023-01-17 06:01:21.348605+00:00) gcworker stdout | 2023-01-17 06:01:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725681349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:01:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:01:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:01:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:01:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:01:22,060 [231] [DEBUG] [app] Starting request: urn:request:76610620-d284-42bf-b549-e12a74ec15d1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:22,060 [235] [DEBUG] [app] Starting request: urn:request:df2ca7ab-545a-49ca-8a8f-2fd55fe11356 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:22,064 [241] [DEBUG] [app] Starting request: urn:request:1f8c44cb-cc54-43f0-aec7-f02af7f9167f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:22,065 [241] [DEBUG] [app] Ending request: urn:request:1f8c44cb-cc54-43f0-aec7-f02af7f9167f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1f8c44cb-cc54-43f0-aec7-f02af7f9167f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:01:22,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:01:22,065 [230] [DEBUG] [app] Starting request: urn:request:82b135d5-2733-4983-9814-1b0c84dd5c05 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:01:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:01:22,066 [230] [DEBUG] [app] Ending request: urn:request:82b135d5-2733-4983-9814-1b0c84dd5c05 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82b135d5-2733-4983-9814-1b0c84dd5c05', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:01:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:22,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:22,069 [236] [DEBUG] [app] Starting request: urn:request:17f24ca8-52f3-4067-b3f1-0eea61f5cb0d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:22,070 [236] [DEBUG] [app] Ending request: urn:request:17f24ca8-52f3-4067-b3f1-0eea61f5cb0d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17f24ca8-52f3-4067-b3f1-0eea61f5cb0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:22,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:22,071 [236] [DEBUG] [app] Starting request: urn:request:90103bee-e5f3-4ae8-bf4b-fb561dea4a53 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:22,071 [236] [DEBUG] [app] Ending request: urn:request:90103bee-e5f3-4ae8-bf4b-fb561dea4a53 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:90103bee-e5f3-4ae8-bf4b-fb561dea4a53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.000 159 0.001) gunicorn-web stdout | 2023-01-17 06:01:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:22,087 [231] [DEBUG] [app] Ending request: urn:request:76610620-d284-42bf-b549-e12a74ec15d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76610620-d284-42bf-b549-e12a74ec15d1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:01:22,088 [235] [DEBUG] [app] Ending request: urn:request:df2ca7ab-545a-49ca-8a8f-2fd55fe11356 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df2ca7ab-545a-49ca-8a8f-2fd55fe11356', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 06:01:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:01:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:37.256652+00:00 (in 13.719194 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:01:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:23 UTC)" (scheduled at 2023-01-17 06:01:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:01:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 1, 23, 537811), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:01:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 23, 537811), True, datetime.datetime(2023, 1, 17, 6, 1, 23, 537811), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:01:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 23, 537811), True, datetime.datetime(2023, 1, 17, 6, 1, 23, 537811), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 1, 23, 537811), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:01:23,555 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:01:23,555 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:35.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 06:01:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:35 UTC)" (scheduled at 2023-01-17 06:01:25.646354+00:00) notificationworker stdout | 2023-01-17 06:01:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 25, 647223), True, datetime.datetime(2023, 1, 17, 6, 1, 25, 647223), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:35 UTC)" executed successfully builder stdout | 2023-01-17 06:01:26,805 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:26,818 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:26,818 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:26,828 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:01:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:01:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:35.853396+00:00 (in 6.889262 seconds) exportactionlogsworker stdout | 2023-01-17 06:01:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:28 UTC)" (scheduled at 2023-01-17 06:01:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:01:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:01:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:30,368 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:01:30,440 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 06:01:31,393 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 06:01:31,476 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:01:31,478 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:01:31,481 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:01:31,483 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:01:31,486 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 06:01:31,863 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:01:31,870 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:01:31,951 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:01:32,468 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:01:35,462 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:01:35,464 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:01:35,467 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 06:01:35,574 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} notificationworker stdout | 2023-01-17 06:01:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:45.646354+00:00 (in 9.999568 seconds) notificationworker stdout | 2023-01-17 06:01:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:45 UTC)" (scheduled at 2023-01-17 06:01:35.646354+00:00) notificationworker stdout | 2023-01-17 06:01:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 35, 647158), True, datetime.datetime(2023, 1, 17, 6, 1, 35, 647158), 0, 'notification/%', 50, 1, 0]) expiredappspecifictokenworker stdout | 2023-01-17 06:01:35,649 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 06:01:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:01:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:01:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:54.963413+00:00 (in 19.109495 seconds) exportactionlogsworker stdout | 2023-01-17 06:01:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:35 UTC)" (scheduled at 2023-01-17 06:01:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:01:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:01:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 35, 854362), True, datetime.datetime(2023, 1, 17, 6, 1, 35, 854362), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:01:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:01:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:01:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:01:35,923 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:01:35,925 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:01:35,929 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:01:35,932 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:01:35,934 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:01:35,936 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:01:35,939 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:01:35,941 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:01:35,944 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:01:36,730 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:01:36,765 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 06:01:36,838 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:36,859 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:36,859 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:36,870 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:37,061 [231] [DEBUG] [app] Starting request: urn:request:594e5f29-3209-44f5-94bc-c3dca4ef0997 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:37,061 [235] [DEBUG] [app] Starting request: urn:request:f4d22e6b-bef8-431a-aaf6-65dce8d8f8d5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:37,066 [230] [DEBUG] [app] Starting request: urn:request:3df57f32-54ed-4a8f-8b57-52abe6ff6aa4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:37,066 [240] [DEBUG] [app] Starting request: urn:request:580059e1-d332-4cc4-bba0-213895018d0e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:37,066 [230] [DEBUG] [app] Ending request: urn:request:3df57f32-54ed-4a8f-8b57-52abe6ff6aa4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3df57f32-54ed-4a8f-8b57-52abe6ff6aa4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:01:37,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:01:37,067 [240] [DEBUG] [app] Ending request: urn:request:580059e1-d332-4cc4-bba0-213895018d0e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:580059e1-d332-4cc4-bba0-213895018d0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:37,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:01:37,067 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:01:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,069 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:37,071 [236] [DEBUG] [app] Starting request: urn:request:3ffd4b42-2b86-4ecc-989d-21a5c1386fd1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:37,071 [235] [DEBUG] [app] Starting request: urn:request:6450d7ac-2953-4431-ba10-6e06e1caa9fb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:37,071 [236] [DEBUG] [app] Ending request: urn:request:3ffd4b42-2b86-4ecc-989d-21a5c1386fd1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ffd4b42-2b86-4ecc-989d-21a5c1386fd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:37,072 [235] [DEBUG] [app] Ending request: urn:request:6450d7ac-2953-4431-ba10-6e06e1caa9fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6450d7ac-2953-4431-ba10-6e06e1caa9fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:01:37,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:37,088 [235] [DEBUG] [app] Ending request: urn:request:f4d22e6b-bef8-431a-aaf6-65dce8d8f8d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f4d22e6b-bef8-431a-aaf6-65dce8d8f8d5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:37,088 [231] [DEBUG] [app] Ending request: urn:request:594e5f29-3209-44f5-94bc-c3dca4ef0997 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:594e5f29-3209-44f5-94bc-c3dca4ef0997', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:37,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:01:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) repositorygcworker stdout | 2023-01-17 06:01:37,106 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} securityscanningnotificationworker stdout | 2023-01-17 06:01:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:01:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:37.537407+00:00 (in 0.280296 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:01:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:37 UTC)" (scheduled at 2023-01-17 06:01:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:01:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:01:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 37, 257539), True, datetime.datetime(2023, 1, 17, 6, 1, 37, 257539), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:01:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:01:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:01:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:01:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:01:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:37.256652+00:00 (in 59.718771 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:01:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:37 UTC)" (scheduled at 2023-01-17 06:01:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:01:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:01:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 06:01:37,656 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:01:37,792 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:01:37,952 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:01:38,099 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:01:38,679 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:01:39,167 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:01:39,249 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:01:39,535 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:01:40,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:01:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:10.738568+00:00 (in 29.999408 seconds) buildlogsarchiver stdout | 2023-01-17 06:01:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:10 UTC)" (scheduled at 2023-01-17 06:01:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:01:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 1, 40, 739520), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:01:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:01:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:01:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:01:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:01:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:42.941536+00:00 (in 1.792855 seconds) namespacegcworker stdout | 2023-01-17 06:01:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:41 UTC)" (scheduled at 2023-01-17 06:01:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:01:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:01:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 41, 149045), True, datetime.datetime(2023, 1, 17, 6, 1, 41, 149045), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:01:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:01:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:01:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:01:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:01:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:06.941185+00:00 (in 23.999179 seconds) namespacegcworker stdout | 2023-01-17 06:01:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:42 UTC)" (scheduled at 2023-01-17 06:01:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:01:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:01:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:02:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:55.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 06:01:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:55 UTC)" (scheduled at 2023-01-17 06:01:45.646354+00:00) notificationworker stdout | 2023-01-17 06:01:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 45, 647228), True, datetime.datetime(2023, 1, 17, 6, 1, 45, 647228), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:01:55 UTC)" executed successfully builder stdout | 2023-01-17 06:01:46,880 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:46,893 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:46,893 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:46,904 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:01:49.537472+00:00 (in 2.600047 seconds) securityworker stdout | 2023-01-17 06:01:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:16 UTC)" (scheduled at 2023-01-17 06:01:46.936988+00:00) securityworker stdout | 2023-01-17 06:01:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:01:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:01:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:16.936988+00:00 (in 27.399037 seconds) securityworker stdout | 2023-01-17 06:01:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:19 UTC)" (scheduled at 2023-01-17 06:01:49.537472+00:00) securityworker stdout | 2023-01-17 06:01:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:01:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:01:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:01:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:01:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:01:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:01:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:01:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:01:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:01:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:01:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:01:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:01:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:01:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:01:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:01:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:21.348605+00:00 (in 29.999510 seconds) gcworker stdout | 2023-01-17 06:01:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:21 UTC)" (scheduled at 2023-01-17 06:01:51.348605+00:00) gcworker stdout | 2023-01-17 06:01:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725711349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:01:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:01:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:01:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:01:52,060 [233] [DEBUG] [app] Starting request: urn:request:0eb8b413-f6b6-4efb-b982-43482b8e4ab4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:52,060 [231] [DEBUG] [app] Starting request: urn:request:e25f5f30-520a-43fe-8dab-0d15bd2dc7e1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:01:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:01:52,065 [241] [DEBUG] [app] Starting request: urn:request:0b667ac5-c6d9-4a60-888a-10ffc638795f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:52,065 [243] [DEBUG] [app] Starting request: urn:request:1675a6ba-fdb9-43d7-9454-b7cea1091777 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:01:52,065 [241] [DEBUG] [app] Ending request: urn:request:0b667ac5-c6d9-4a60-888a-10ffc638795f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b667ac5-c6d9-4a60-888a-10ffc638795f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:01:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:01:52,066 [243] [DEBUG] [app] Ending request: urn:request:1675a6ba-fdb9-43d7-9454-b7cea1091777 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1675a6ba-fdb9-43d7-9454-b7cea1091777', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:01:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:01:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:01:52,070 [235] [DEBUG] [app] Starting request: urn:request:deeb2b86-957d-4222-8e37-157eba07f083 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:52,070 [235] [DEBUG] [app] Ending request: urn:request:deeb2b86-957d-4222-8e37-157eba07f083 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:deeb2b86-957d-4222-8e37-157eba07f083', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:52,070 [236] [DEBUG] [app] Starting request: urn:request:82ed0153-9afd-48b6-82a8-3c1541e5645c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:01:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:01:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:01:52,071 [236] [DEBUG] [app] Ending request: urn:request:82ed0153-9afd-48b6-82a8-3c1541e5645c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82ed0153-9afd-48b6-82a8-3c1541e5645c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:01:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:01:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:01:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:01:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:01:52,087 [231] [DEBUG] [app] Ending request: urn:request:e25f5f30-520a-43fe-8dab-0d15bd2dc7e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e25f5f30-520a-43fe-8dab-0d15bd2dc7e1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:52,087 [233] [DEBUG] [app] Ending request: urn:request:0eb8b413-f6b6-4efb-b982-43482b8e4ab4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0eb8b413-f6b6-4efb-b982-43482b8e4ab4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:01:52,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:01:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:01:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:01:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) exportactionlogsworker stdout | 2023-01-17 06:01:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:01:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:28.963662+00:00 (in 33.999783 seconds) exportactionlogsworker stdout | 2023-01-17 06:01:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:54 UTC)" (scheduled at 2023-01-17 06:01:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 06:01:54,965 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 1, 54, 964358), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:01:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 54, 964358), True, datetime.datetime(2023, 1, 17, 6, 1, 54, 964358), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:01:54,978 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 1, 54, 964358), True, datetime.datetime(2023, 1, 17, 6, 1, 54, 964358), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 1, 54, 964358), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:01:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:01:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:06:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:01:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:01:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:01.636362+00:00 (in 5.989544 seconds) notificationworker stdout | 2023-01-17 06:01:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:05 UTC)" (scheduled at 2023-01-17 06:01:55.646354+00:00) notificationworker stdout | 2023-01-17 06:01:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:01:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 1, 55, 647227), True, datetime.datetime(2023, 1, 17, 6, 1, 55, 647227), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:01:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:01:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:01:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:05 UTC)" executed successfully builder stdout | 2023-01-17 06:01:56,914 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:01:56,927 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:01:56,927 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:01:56,938 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:00,379 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:02:00,475 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 06:02:01,429 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 06:02:01,483 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:02:01,485 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:02:01,488 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:02:01,490 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:02:01,493 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:02:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:05.646354+00:00 (in 4.009555 seconds) notificationworker stdout | 2023-01-17 06:02:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:01 UTC)" (scheduled at 2023-01-17 06:02:01.636362+00:00) notificationworker stdout | 2023-01-17 06:02:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:02:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:01 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:02:01,879 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:02:01,904 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:02:01,978 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:02:02,504 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:02:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:02:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:19.346120+00:00 (in 14.408856 seconds) repositorygcworker stdout | 2023-01-17 06:02:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:04 UTC)" (scheduled at 2023-01-17 06:02:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:02:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:02:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 4, 937694), True, datetime.datetime(2023, 1, 17, 6, 2, 4, 937694), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:02:04,948 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:02:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:02:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:02:05,469 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:02:05,471 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:02:05,474 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 06:02:05,590 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} notificationworker stdout | 2023-01-17 06:02:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:15.646354+00:00 (in 9.999586 seconds) notificationworker stdout | 2023-01-17 06:02:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:15 UTC)" (scheduled at 2023-01-17 06:02:05.646354+00:00) notificationworker stdout | 2023-01-17 06:02:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 5, 647151), True, datetime.datetime(2023, 1, 17, 6, 2, 5, 647151), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:15 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 06:02:05,683 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:02:05,931 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:02:05,933 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:02:05,938 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:02:05,940 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:02:05,946 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:02:05,948 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:02:05,950 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:02:05,953 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:02:05,955 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:02:06,748 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:02:06,799 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} namespacegcworker stdout | 2023-01-17 06:02:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:02:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:41.148245+00:00 (in 34.206578 seconds) namespacegcworker stdout | 2023-01-17 06:02:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:07:06 UTC)" (scheduled at 2023-01-17 06:02:06.941185+00:00) namespacegcworker stdout | 2023-01-17 06:02:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 2, 6, 942038), 'namespacegc/%']) builder stdout | 2023-01-17 06:02:06,943 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue namespacegcworker stdout | 2023-01-17 06:02:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 2, 6, 942038), True, datetime.datetime(2023, 1, 17, 6, 2, 6, 942038), 0, 'namespacegc/%']) builder stdout | 2023-01-17 06:02:06,954 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:06,954 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics namespacegcworker stdout | 2023-01-17 06:02:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 2, 6, 942038), True, datetime.datetime(2023, 1, 17, 6, 2, 6, 942038), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 2, 6, 942038), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:02:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:02:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:07:06 UTC)" executed successfully builder stdout | 2023-01-17 06:02:06,965 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:07,060 [231] [DEBUG] [app] Starting request: urn:request:e458496e-f775-4aea-9c53-7c282ab3094c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:07,060 [235] [DEBUG] [app] Starting request: urn:request:97514701-2f76-417f-80e4-2f6209323fa0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:07,065 [230] [DEBUG] [app] Starting request: urn:request:671e14ae-9f32-401a-b96a-0c362b4570dc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:07,065 [243] [DEBUG] [app] Starting request: urn:request:8baf14f5-bebd-49a1-a592-19d3dc8daa62 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:07,065 [230] [DEBUG] [app] Ending request: urn:request:671e14ae-9f32-401a-b96a-0c362b4570dc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:671e14ae-9f32-401a-b96a-0c362b4570dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:02:07,066 [243] [DEBUG] [app] Ending request: urn:request:8baf14f5-bebd-49a1-a592-19d3dc8daa62 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8baf14f5-bebd-49a1-a592-19d3dc8daa62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:02:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:02:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:07,070 [235] [DEBUG] [app] Starting request: urn:request:46d7ede7-f8a9-4766-8ebd-f3c48f0e9b34 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:07,070 [235] [DEBUG] [app] Ending request: urn:request:46d7ede7-f8a9-4766-8ebd-f3c48f0e9b34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:46d7ede7-f8a9-4766-8ebd-f3c48f0e9b34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:07,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:07,071 [233] [DEBUG] [app] Starting request: urn:request:c195c639-67f9-48c8-b075-25fcbb1df5e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:07,072 [233] [DEBUG] [app] Ending request: urn:request:c195c639-67f9-48c8-b075-25fcbb1df5e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c195c639-67f9-48c8-b075-25fcbb1df5e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:02:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:07,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:07,086 [231] [DEBUG] [app] Ending request: urn:request:e458496e-f775-4aea-9c53-7c282ab3094c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e458496e-f775-4aea-9c53-7c282ab3094c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:07,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:07,086 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.027) gunicorn-web stdout | 2023-01-17 06:02:07,088 [235] [DEBUG] [app] Ending request: urn:request:97514701-2f76-417f-80e4-2f6209323fa0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97514701-2f76-417f-80e4-2f6209323fa0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) repositorygcworker stdout | 2023-01-17 06:02:07,143 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:02:07,691 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:02:07,827 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:02:07,969 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:02:08,134 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:02:08,687 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:02:09,177 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:02:09,283 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:02:09,556 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:02:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:02:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:40.738568+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 06:02:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:40 UTC)" (scheduled at 2023-01-17 06:02:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:02:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 2, 10, 739393), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:02:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:02:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:02:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:02:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:25.646354+00:00 (in 9.999594 seconds) notificationworker stdout | 2023-01-17 06:02:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:25 UTC)" (scheduled at 2023-01-17 06:02:15.646354+00:00) notificationworker stdout | 2023-01-17 06:02:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 15, 647148), True, datetime.datetime(2023, 1, 17, 6, 2, 15, 647148), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:02:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:19.537472+00:00 (in 2.600002 seconds) securityworker stdout | 2023-01-17 06:02:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:46 UTC)" (scheduled at 2023-01-17 06:02:16.936988+00:00) securityworker stdout | 2023-01-17 06:02:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:16,953 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:16,956 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:16,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:46 UTC)" executed successfully builder stdout | 2023-01-17 06:02:16,975 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:16,987 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:16,987 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:16,997 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:02:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:02:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:04.936835+00:00 (in 45.590249 seconds) repositorygcworker stdout | 2023-01-17 06:02:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:19 UTC)" (scheduled at 2023-01-17 06:02:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:02:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:02:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:02:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:46.936988+00:00 (in 27.399007 seconds) securityworker stdout | 2023-01-17 06:02:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:49 UTC)" (scheduled at 2023-01-17 06:02:19.537472+00:00) securityworker stdout | 2023-01-17 06:02:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:02:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:02:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:02:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:02:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:02:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:02:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:02:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:02:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:51.348605+00:00 (in 29.999509 seconds) gcworker stdout | 2023-01-17 06:02:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:51 UTC)" (scheduled at 2023-01-17 06:02:21.348605+00:00) gcworker stdout | 2023-01-17 06:02:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725741349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:02:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:02:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:02:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:02:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:02:22,060 [231] [DEBUG] [app] Starting request: urn:request:0f929e56-68c0-4e47-8711-79b0e7f4b25f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:22,060 [235] [DEBUG] [app] Starting request: urn:request:6b29e417-5d52-42b8-9d23-ee3b65699b55 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:22,065 [230] [DEBUG] [app] Starting request: urn:request:7fa9cd16-df93-40d2-a153-3e4865abfb2d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:22,065 [243] [DEBUG] [app] Starting request: urn:request:96aad2ac-b5e3-4b7f-b155-262d928c572f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:22,066 [230] [DEBUG] [app] Ending request: urn:request:7fa9cd16-df93-40d2-a153-3e4865abfb2d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7fa9cd16-df93-40d2-a153-3e4865abfb2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:02:22,066 [243] [DEBUG] [app] Ending request: urn:request:96aad2ac-b5e3-4b7f-b155-262d928c572f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96aad2ac-b5e3-4b7f-b155-262d928c572f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:02:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:02:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:02:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:22,070 [231] [DEBUG] [app] Starting request: urn:request:7f6c6dba-0291-456f-9df3-e155a1a5b109 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:22,070 [231] [DEBUG] [app] Ending request: urn:request:7f6c6dba-0291-456f-9df3-e155a1a5b109 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f6c6dba-0291-456f-9df3-e155a1a5b109', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:22,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:22,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:22,072 [236] [DEBUG] [app] Starting request: urn:request:067071be-db1b-42bd-928b-06b452e49483 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:22,072 [236] [DEBUG] [app] Ending request: urn:request:067071be-db1b-42bd-928b-06b452e49483 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:067071be-db1b-42bd-928b-06b452e49483', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:22,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:22,087 [235] [DEBUG] [app] Ending request: urn:request:6b29e417-5d52-42b8-9d23-ee3b65699b55 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b29e417-5d52-42b8-9d23-ee3b65699b55', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:22,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:02:22,093 [231] [DEBUG] [app] Ending request: urn:request:0f929e56-68c0-4e47-8711-79b0e7f4b25f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f929e56-68c0-4e47-8711-79b0e7f4b25f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:22,093 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:22,093 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.034 47 0.034) notificationworker stdout | 2023-01-17 06:02:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:35.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 06:02:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:35 UTC)" (scheduled at 2023-01-17 06:02:25.646354+00:00) notificationworker stdout | 2023-01-17 06:02:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 25, 647233), True, datetime.datetime(2023, 1, 17, 6, 2, 25, 647233), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:35 UTC)" executed successfully builder stdout | 2023-01-17 06:02:27,008 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:27,020 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:27,020 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:27,031 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:02:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:02:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:35.853396+00:00 (in 6.889189 seconds) exportactionlogsworker stdout | 2023-01-17 06:02:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:28 UTC)" (scheduled at 2023-01-17 06:02:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:02:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:02:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:02:30,414 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:02:30,509 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} queuecleanupworker stdout | 2023-01-17 06:02:31,465 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 06:02:31,488 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:02:31,491 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:02:31,494 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:02:31,497 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:02:31,499 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} logrotateworker stdout | 2023-01-17 06:02:31,888 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:02:31,933 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:02:32,013 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:02:32,513 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:02:35,477 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:02:35,479 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:02:35,481 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} buildlogsarchiver stdout | 2023-01-17 06:02:35,625 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} notificationworker stdout | 2023-01-17 06:02:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:45.646354+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 06:02:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:45 UTC)" (scheduled at 2023-01-17 06:02:35.646354+00:00) notificationworker stdout | 2023-01-17 06:02:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 35, 647185), True, datetime.datetime(2023, 1, 17, 6, 2, 35, 647185), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:45 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 06:02:35,718 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} exportactionlogsworker stdout | 2023-01-17 06:02:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:02:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:28.963662+00:00 (in 53.109761 seconds) exportactionlogsworker stdout | 2023-01-17 06:02:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:35 UTC)" (scheduled at 2023-01-17 06:02:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:02:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:02:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 35, 854296), True, datetime.datetime(2023, 1, 17, 6, 2, 35, 854296), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:02:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:02:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:02:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:02:35,942 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:02:35,944 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:02:35,949 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:02:35,951 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:02:35,954 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:02:35,956 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:02:35,959 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:02:35,962 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:02:35,965 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:02:36,757 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:02:36,834 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 06:02:37,041 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:37,054 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:37,054 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics gunicorn-web stdout | 2023-01-17 06:02:37,060 [231] [DEBUG] [app] Starting request: urn:request:6a55cc95-6137-490a-b953-af6d20c61cdb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:37,060 [236] [DEBUG] [app] Starting request: urn:request:220eb95c-e567-40b4-b16b-ff4d51551840 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:37,065 [230] [DEBUG] [app] Starting request: urn:request:c64aaffc-b535-4b55-8838-8ddbb6ba01df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} builder stdout | 2023-01-17 06:02:37,065 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-registry stdout | 2023-01-17 06:02:37,065 [230] [DEBUG] [app] Ending request: urn:request:c64aaffc-b535-4b55-8838-8ddbb6ba01df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c64aaffc-b535-4b55-8838-8ddbb6ba01df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:37,065 [243] [DEBUG] [app] Starting request: urn:request:74e9ab6c-3173-49d8-9cf3-e11be77eff21 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:02:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:02:37,065 [243] [DEBUG] [app] Ending request: urn:request:74e9ab6c-3173-49d8-9cf3-e11be77eff21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:74e9ab6c-3173-49d8-9cf3-e11be77eff21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:37,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:02:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:02:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:37,070 [235] [DEBUG] [app] Starting request: urn:request:c97d8402-aee0-44bf-a6bf-34ea743fab6e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:37,070 [233] [DEBUG] [app] Starting request: urn:request:810cfc7d-8b49-4573-aad1-4f63eaff1b7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:37,070 [235] [DEBUG] [app] Ending request: urn:request:c97d8402-aee0-44bf-a6bf-34ea743fab6e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c97d8402-aee0-44bf-a6bf-34ea743fab6e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:37,070 [233] [DEBUG] [app] Ending request: urn:request:810cfc7d-8b49-4573-aad1-4f63eaff1b7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:810cfc7d-8b49-4573-aad1-4f63eaff1b7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:02:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:37,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:37,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:37,086 [231] [DEBUG] [app] Ending request: urn:request:6a55cc95-6137-490a-b953-af6d20c61cdb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a55cc95-6137-490a-b953-af6d20c61cdb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:37,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:37,086 [236] [DEBUG] [app] Ending request: urn:request:220eb95c-e567-40b4-b16b-ff4d51551840 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:220eb95c-e567-40b4-b16b-ff4d51551840', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:37,086 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.027) gunicorn-web stdout | 2023-01-17 06:02:37,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" repositorygcworker stdout | 2023-01-17 06:02:37,178 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} securityscanningnotificationworker stdout | 2023-01-17 06:02:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:02:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:37.537407+00:00 (in 0.280357 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:02:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:37 UTC)" (scheduled at 2023-01-17 06:02:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:02:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:02:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 37, 257445), True, datetime.datetime(2023, 1, 17, 6, 2, 37, 257445), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:02:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:02:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:02:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:02:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:02:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:37.256652+00:00 (in 59.718867 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:02:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:37 UTC)" (scheduled at 2023-01-17 06:02:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:02:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:02:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 06:02:37,727 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:02:37,851 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:02:38,004 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:02:38,170 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:02:38,703 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:02:39,183 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:02:39,313 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:02:39,591 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:02:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:02:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:10.738568+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 06:02:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:10 UTC)" (scheduled at 2023-01-17 06:02:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:02:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 2, 40, 739416), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:02:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:02:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:02:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:02:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:02:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:42.941536+00:00 (in 1.792817 seconds) namespacegcworker stdout | 2023-01-17 06:02:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:41 UTC)" (scheduled at 2023-01-17 06:02:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:02:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:02:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 41, 149124), True, datetime.datetime(2023, 1, 17, 6, 2, 41, 149124), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:02:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:02:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:02:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:02:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:02:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:41.148245+00:00 (in 58.206243 seconds) namespacegcworker stdout | 2023-01-17 06:02:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:42 UTC)" (scheduled at 2023-01-17 06:02:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:02:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:02:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:03:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:02:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:55.646354+00:00 (in 9.999563 seconds) notificationworker stdout | 2023-01-17 06:02:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:55 UTC)" (scheduled at 2023-01-17 06:02:45.646354+00:00) notificationworker stdout | 2023-01-17 06:02:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 45, 647172), True, datetime.datetime(2023, 1, 17, 6, 2, 45, 647172), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:45,668 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:45,668 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:45,668 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:02:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:02:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:02:49.537472+00:00 (in 2.599998 seconds) securityworker stdout | 2023-01-17 06:02:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:16 UTC)" (scheduled at 2023-01-17 06:02:46.936988+00:00) securityworker stdout | 2023-01-17 06:02:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:16 UTC)" executed successfully builder stdout | 2023-01-17 06:02:47,075 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:47,087 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:47,087 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:47,098 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:02:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:16.936988+00:00 (in 27.399023 seconds) securityworker stdout | 2023-01-17 06:02:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:19 UTC)" (scheduled at 2023-01-17 06:02:49.537472+00:00) securityworker stdout | 2023-01-17 06:02:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:02:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:02:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:02:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:02:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:02:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:02:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:02:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:02:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:02:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:02:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:02:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:02:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:02:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:02:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:02:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:21.348605+00:00 (in 29.999505 seconds) gcworker stdout | 2023-01-17 06:02:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:21 UTC)" (scheduled at 2023-01-17 06:02:51.348605+00:00) gcworker stdout | 2023-01-17 06:02:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725771349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:02:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:02:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:02:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:02:52,060 [231] [DEBUG] [app] Starting request: urn:request:7526dfc9-eb84-4af9-91fe-628fad154be8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:52,060 [233] [DEBUG] [app] Starting request: urn:request:b2594b4a-3ae6-47c7-a472-39d548fc7f1b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:02:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:02:52,065 [242] [DEBUG] [app] Starting request: urn:request:7cde1261-29e7-41f4-a502-98006d6e8966 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:52,065 [240] [DEBUG] [app] Starting request: urn:request:be52bb0d-dbc9-4342-b7a4-819083966f30 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:02:52,065 [242] [DEBUG] [app] Ending request: urn:request:7cde1261-29e7-41f4-a502-98006d6e8966 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7cde1261-29e7-41f4-a502-98006d6e8966', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:52,065 [240] [DEBUG] [app] Ending request: urn:request:be52bb0d-dbc9-4342-b7a4-819083966f30 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be52bb0d-dbc9-4342-b7a4-819083966f30', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:02:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:02:52,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:02:52,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:02:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:02:52,070 [233] [DEBUG] [app] Starting request: urn:request:f7ac9b17-8088-40cb-b7a3-7399a791adcd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:52,070 [236] [DEBUG] [app] Starting request: urn:request:7513340e-8414-4589-8cd0-f0b7519a5334 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:02:52,071 [233] [DEBUG] [app] Ending request: urn:request:f7ac9b17-8088-40cb-b7a3-7399a791adcd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7ac9b17-8088-40cb-b7a3-7399a791adcd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:52,071 [236] [DEBUG] [app] Ending request: urn:request:7513340e-8414-4589-8cd0-f0b7519a5334 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7513340e-8414-4589-8cd0-f0b7519a5334', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:02:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:02:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:02:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:02:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:02:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:02:52,087 [233] [DEBUG] [app] Ending request: urn:request:b2594b4a-3ae6-47c7-a472-39d548fc7f1b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2594b4a-3ae6-47c7-a472-39d548fc7f1b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:52,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:52,087 [231] [DEBUG] [app] Ending request: urn:request:7526dfc9-eb84-4af9-91fe-628fad154be8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7526dfc9-eb84-4af9-91fe-628fad154be8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:02:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:02:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:02:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:02:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 06:02:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:02:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:01.636362+00:00 (in 5.989573 seconds) notificationworker stdout | 2023-01-17 06:02:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:05 UTC)" (scheduled at 2023-01-17 06:02:55.646354+00:00) notificationworker stdout | 2023-01-17 06:02:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:02:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 2, 55, 647147), True, datetime.datetime(2023, 1, 17, 6, 2, 55, 647147), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:02:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:02:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:02:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:05 UTC)" executed successfully builder stdout | 2023-01-17 06:02:57,106 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:02:57,118 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:02:57,118 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:02:57,129 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:00,438 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:03:00,516 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:03:01,496 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:03:01,499 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} queuecleanupworker stdout | 2023-01-17 06:03:01,501 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} gunicorn-web stdout | 2023-01-17 06:03:01,504 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:03:01,506 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:03:01,508 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:03:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:05.646354+00:00 (in 4.009558 seconds) notificationworker stdout | 2023-01-17 06:03:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:01 UTC)" (scheduled at 2023-01-17 06:03:01.636362+00:00) notificationworker stdout | 2023-01-17 06:03:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:03:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:01 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:03:01,910 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:03:01,967 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:03:02,036 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:03:02,549 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:03:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:03:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:19.346120+00:00 (in 14.408846 seconds) repositorygcworker stdout | 2023-01-17 06:03:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:04 UTC)" (scheduled at 2023-01-17 06:03:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:03:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:03:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 4, 937688), True, datetime.datetime(2023, 1, 17, 6, 3, 4, 937688), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:03:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:03:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:03:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:03:05,484 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:03:05,487 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:03:05,489 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:03:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:15.646354+00:00 (in 9.999550 seconds) notificationworker stdout | 2023-01-17 06:03:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:15 UTC)" (scheduled at 2023-01-17 06:03:05.646354+00:00) notificationworker stdout | 2023-01-17 06:03:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 5, 647179), True, datetime.datetime(2023, 1, 17, 6, 3, 5, 647179), 0, 'notification/%', 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:03:05,649 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} notificationworker stdout | 2023-01-17 06:03:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:15 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 06:03:05,729 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:03:05,949 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:03:05,954 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:03:05,959 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:03:05,962 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:03:05,964 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:03:05,967 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:03:05,969 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:03:05,972 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:03:05,975 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:03:06,793 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:03:06,853 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:03:07,060 [231] [DEBUG] [app] Starting request: urn:request:145b0d59-774c-42fc-89db-9888ad8b94c7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:07,060 [236] [DEBUG] [app] Starting request: urn:request:dc303181-cd4f-41fa-9753-51b7857c5423 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:07,065 [242] [DEBUG] [app] Starting request: urn:request:c748fb93-0012-4c81-9dbd-c05263ab46b6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:07,066 [242] [DEBUG] [app] Ending request: urn:request:c748fb93-0012-4c81-9dbd-c05263ab46b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c748fb93-0012-4c81-9dbd-c05263ab46b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:07,065 [237] [DEBUG] [app] Starting request: urn:request:9242b8c3-5f15-40ad-a45a-20eb92754b7f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:03:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:03:07,066 [237] [DEBUG] [app] Ending request: urn:request:9242b8c3-5f15-40ad-a45a-20eb92754b7f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9242b8c3-5f15-40ad-a45a-20eb92754b7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:03:07,066 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:07,070 [235] [DEBUG] [app] Starting request: urn:request:9b2a2508-0ef3-4c21-82c3-35b14d40015c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:07,070 [235] [DEBUG] [app] Ending request: urn:request:9b2a2508-0ef3-4c21-82c3-35b14d40015c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b2a2508-0ef3-4c21-82c3-35b14d40015c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:07,071 [233] [DEBUG] [app] Starting request: urn:request:ea8b11a0-50e6-49e6-bea1-bfa32215589b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:07,071 [233] [DEBUG] [app] Ending request: urn:request:ea8b11a0-50e6-49e6-bea1-bfa32215589b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ea8b11a0-50e6-49e6-bea1-bfa32215589b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:07,087 [236] [DEBUG] [app] Ending request: urn:request:dc303181-cd4f-41fa-9753-51b7857c5423 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc303181-cd4f-41fa-9753-51b7857c5423', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:07,087 [231] [DEBUG] [app] Ending request: urn:request:145b0d59-774c-42fc-89db-9888ad8b94c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:145b0d59-774c-42fc-89db-9888ad8b94c7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:03:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 06:03:07,139 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:07,151 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:07,151 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:07,162 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:03:07,193 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:03:07,762 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:03:07,880 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:03:08,023 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:03:08,182 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:03:08,728 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:03:09,219 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:03:09,339 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:03:09,627 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:03:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:03:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:40.738568+00:00 (in 29.999548 seconds) buildlogsarchiver stdout | 2023-01-17 06:03:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:40 UTC)" (scheduled at 2023-01-17 06:03:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:03:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 3, 10, 739359), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:03:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:03:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:03:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:03:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:25.646354+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 06:03:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:25 UTC)" (scheduled at 2023-01-17 06:03:15.646354+00:00) notificationworker stdout | 2023-01-17 06:03:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 15, 647242), True, datetime.datetime(2023, 1, 17, 6, 3, 15, 647242), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:03:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:19.537472+00:00 (in 2.599977 seconds) securityworker stdout | 2023-01-17 06:03:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:46 UTC)" (scheduled at 2023-01-17 06:03:16.936988+00:00) securityworker stdout | 2023-01-17 06:03:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:46 UTC)" executed successfully builder stdout | 2023-01-17 06:03:17,172 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:17,184 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:17,185 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:17,196 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:03:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:03:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:04.936835+00:00 (in 45.590283 seconds) repositorygcworker stdout | 2023-01-17 06:03:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:19 UTC)" (scheduled at 2023-01-17 06:03:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:03:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:03:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:03:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:46.936988+00:00 (in 27.399040 seconds) securityworker stdout | 2023-01-17 06:03:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:49 UTC)" (scheduled at 2023-01-17 06:03:19.537472+00:00) securityworker stdout | 2023-01-17 06:03:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:03:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:03:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:19,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:19,540 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:03:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:03:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:03:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:03:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:03:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:03:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:51.348605+00:00 (in 29.999539 seconds) gcworker stdout | 2023-01-17 06:03:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:51 UTC)" (scheduled at 2023-01-17 06:03:21.348605+00:00) gcworker stdout | 2023-01-17 06:03:21,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 06:03:21,360 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725801359, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:03:21,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:03:21,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:03:21,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:03:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:03:22,060 [231] [DEBUG] [app] Starting request: urn:request:1bfd397c-ff26-44a0-bc20-351b02701b2f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:22,060 [236] [DEBUG] [app] Starting request: urn:request:3ee8734f-c57a-44b2-8a8f-3f688f40155d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:22,065 [230] [DEBUG] [app] Starting request: urn:request:7fe6e6dc-dfb2-4cf2-93d5-18ea13fcbbd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:22,065 [243] [DEBUG] [app] Starting request: urn:request:6f3410f6-11e8-46b6-b0ed-7ca0cb1afd68 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:22,065 [230] [DEBUG] [app] Ending request: urn:request:7fe6e6dc-dfb2-4cf2-93d5-18ea13fcbbd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7fe6e6dc-dfb2-4cf2-93d5-18ea13fcbbd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:03:22,066 [243] [DEBUG] [app] Ending request: urn:request:6f3410f6-11e8-46b6-b0ed-7ca0cb1afd68 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f3410f6-11e8-46b6-b0ed-7ca0cb1afd68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:03:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:03:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:22,070 [233] [DEBUG] [app] Starting request: urn:request:b23588f1-2161-4def-bb2a-ed020ecf1a47 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:22,070 [233] [DEBUG] [app] Ending request: urn:request:b23588f1-2161-4def-bb2a-ed020ecf1a47 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b23588f1-2161-4def-bb2a-ed020ecf1a47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:22,070 [235] [DEBUG] [app] Starting request: urn:request:ab52bcec-6878-41ac-a714-dbbbee0434b4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:22,071 [235] [DEBUG] [app] Ending request: urn:request:ab52bcec-6878-41ac-a714-dbbbee0434b4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ab52bcec-6878-41ac-a714-dbbbee0434b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:22,087 [236] [DEBUG] [app] Ending request: urn:request:3ee8734f-c57a-44b2-8a8f-3f688f40155d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ee8734f-c57a-44b2-8a8f-3f688f40155d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:22,087 [231] [DEBUG] [app] Ending request: urn:request:1bfd397c-ff26-44a0-bc20-351b02701b2f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1bfd397c-ff26-44a0-bc20-351b02701b2f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:03:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) notificationworker stdout | 2023-01-17 06:03:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:35.646354+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 06:03:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:35 UTC)" (scheduled at 2023-01-17 06:03:25.646354+00:00) notificationworker stdout | 2023-01-17 06:03:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 25, 647247), True, datetime.datetime(2023, 1, 17, 6, 3, 25, 647247), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:35 UTC)" executed successfully builder stdout | 2023-01-17 06:03:27,206 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:27,219 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:27,219 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:27,230 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:03:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:03:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:35.853396+00:00 (in 6.889276 seconds) exportactionlogsworker stdout | 2023-01-17 06:03:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:28 UTC)" (scheduled at 2023-01-17 06:03:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:03:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:03:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:03:30,474 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:03:30,549 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:03:31,503 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:03:31,505 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:03:31,510 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:03:31,512 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:03:31,515 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:03:31,534 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:03:31,927 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:03:32,000 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:03:32,052 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:03:32,575 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:03:35,492 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:03:35,495 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:03:35,497 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:03:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:36.635997+00:00 (in 0.989169 seconds) notificationworker stdout | 2023-01-17 06:03:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:45 UTC)" (scheduled at 2023-01-17 06:03:35.646354+00:00) notificationworker stdout | 2023-01-17 06:03:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 35, 647238), True, datetime.datetime(2023, 1, 17, 6, 3, 35, 647238), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:45 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:03:35,685 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:03:35,764 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} exportactionlogsworker stdout | 2023-01-17 06:03:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:03:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:28.963662+00:00 (in 53.109755 seconds) exportactionlogsworker stdout | 2023-01-17 06:03:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:35 UTC)" (scheduled at 2023-01-17 06:03:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:03:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:03:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 35, 854324), True, datetime.datetime(2023, 1, 17, 6, 3, 35, 854324), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:03:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:03:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:03:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:03:35,956 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:03:35,963 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:03:35,969 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:03:35,971 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:03:35,974 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:03:35,977 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:03:35,980 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:03:35,982 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:03:35,984 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} notificationworker stdout | 2023-01-17 06:03:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:45.646354+00:00 (in 9.009884 seconds) notificationworker stdout | 2023-01-17 06:03:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:08:36 UTC)" (scheduled at 2023-01-17 06:03:36.635997+00:00) notificationworker stdout | 2023-01-17 06:03:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 3, 36, 636835), 'notification/%']) notificationworker stdout | 2023-01-17 06:03:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 3, 36, 636835), True, datetime.datetime(2023, 1, 17, 6, 3, 36, 636835), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:03:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 3, 36, 636835), True, datetime.datetime(2023, 1, 17, 6, 3, 36, 636835), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 3, 36, 636835), 'notification/%']) notificationworker stdout | 2023-01-17 06:03:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:08:36 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:03:36,812 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:03:36,870 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:03:37,060 [236] [DEBUG] [app] Starting request: urn:request:b60560d8-5736-4a38-932b-6924607daca0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:37,060 [231] [DEBUG] [app] Starting request: urn:request:29293411-2480-4cfc-a1cb-5e60f09a7acb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:37,064 [230] [DEBUG] [app] Starting request: urn:request:31a68dc1-233d-4e8c-a241-598f64b2b2eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:37,065 [243] [DEBUG] [app] Starting request: urn:request:7b79ca70-b8ab-43f5-8bab-8dfa60b15450 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:37,065 [230] [DEBUG] [app] Ending request: urn:request:31a68dc1-233d-4e8c-a241-598f64b2b2eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31a68dc1-233d-4e8c-a241-598f64b2b2eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:03:37,065 [243] [DEBUG] [app] Ending request: urn:request:7b79ca70-b8ab-43f5-8bab-8dfa60b15450 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b79ca70-b8ab-43f5-8bab-8dfa60b15450', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:03:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:03:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:37,070 [235] [DEBUG] [app] Starting request: urn:request:e316e464-44bf-4d88-b2bf-84dcfdbc3115 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:37,070 [235] [DEBUG] [app] Ending request: urn:request:e316e464-44bf-4d88-b2bf-84dcfdbc3115 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e316e464-44bf-4d88-b2bf-84dcfdbc3115', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:37,070 [233] [DEBUG] [app] Starting request: urn:request:8eb7310b-3e2c-4025-bb41-5c76cbc08c43 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:37,071 [233] [DEBUG] [app] Ending request: urn:request:8eb7310b-3e2c-4025-bb41-5c76cbc08c43 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8eb7310b-3e2c-4025-bb41-5c76cbc08c43', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:03:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:37,087 [236] [DEBUG] [app] Ending request: urn:request:b60560d8-5736-4a38-932b-6924607daca0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b60560d8-5736-4a38-932b-6924607daca0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:37,087 [231] [DEBUG] [app] Ending request: urn:request:29293411-2480-4cfc-a1cb-5e60f09a7acb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29293411-2480-4cfc-a1cb-5e60f09a7acb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 06:03:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" repositorygcworker stdout | 2023-01-17 06:03:37,209 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 06:03:37,240 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:37,251 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:37,252 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics securityscanningnotificationworker stdout | 2023-01-17 06:03:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:03:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:37.537407+00:00 (in 0.280415 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:03:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:37 UTC)" (scheduled at 2023-01-17 06:03:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:03:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:03:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 37, 257362), True, datetime.datetime(2023, 1, 17, 6, 3, 37, 257362), 0, 'secscanv4/%', 50, 1, 0]) builder stdout | 2023-01-17 06:03:37,262 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:03:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:03:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:03:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:03:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:03:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:37.256652+00:00 (in 59.718879 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:03:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:37 UTC)" (scheduled at 2023-01-17 06:03:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:03:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:03:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 06:03:37,771 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:03:37,892 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:03:38,059 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:03:38,198 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:03:38,764 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:03:39,227 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:03:39,365 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:03:39,653 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:03:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:03:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:10.738568+00:00 (in 29.999533 seconds) buildlogsarchiver stdout | 2023-01-17 06:03:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:10 UTC)" (scheduled at 2023-01-17 06:03:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:03:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 3, 40, 739405), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:03:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:03:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:03:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:03:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:03:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:42.941536+00:00 (in 1.792778 seconds) namespacegcworker stdout | 2023-01-17 06:03:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:41 UTC)" (scheduled at 2023-01-17 06:03:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:03:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:03:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 41, 149196), True, datetime.datetime(2023, 1, 17, 6, 3, 41, 149196), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:03:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:03:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:03:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:03:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:03:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:41.148245+00:00 (in 58.206228 seconds) namespacegcworker stdout | 2023-01-17 06:03:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:42 UTC)" (scheduled at 2023-01-17 06:03:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:03:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:03:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:04:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:03:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:55.646354+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 06:03:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:55 UTC)" (scheduled at 2023-01-17 06:03:45.646354+00:00) notificationworker stdout | 2023-01-17 06:03:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 45, 647233), True, datetime.datetime(2023, 1, 17, 6, 3, 45, 647233), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:03:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:03:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:03:49.537472+00:00 (in 2.600030 seconds) securityworker stdout | 2023-01-17 06:03:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:16 UTC)" (scheduled at 2023-01-17 06:03:46.936988+00:00) securityworker stdout | 2023-01-17 06:03:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:16 UTC)" executed successfully builder stdout | 2023-01-17 06:03:47,272 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:47,284 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:47,284 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:47,295 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:03:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:16.936988+00:00 (in 27.399076 seconds) securityworker stdout | 2023-01-17 06:03:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:19 UTC)" (scheduled at 2023-01-17 06:03:49.537472+00:00) securityworker stdout | 2023-01-17 06:03:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:03:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:03:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:03:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:03:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:03:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:03:49,563 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:49,563 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:03:49,563 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:03:49,563 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:03:49,563 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:03:49,565 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:03:49,565 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:03:49,565 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:03:49,565 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:03:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:03:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:21.348605+00:00 (in 29.999553 seconds) gcworker stdout | 2023-01-17 06:03:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:21 UTC)" (scheduled at 2023-01-17 06:03:51.348605+00:00) gcworker stdout | 2023-01-17 06:03:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725831349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:03:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:03:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:03:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:03:52,060 [231] [DEBUG] [app] Starting request: urn:request:0ffb9584-845e-401f-88ca-2e2c13257fd6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:52,060 [236] [DEBUG] [app] Starting request: urn:request:2d709374-258f-4aaf-81bb-2e3e198ed387 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:03:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:52,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:03:52,065 [230] [DEBUG] [app] Starting request: urn:request:c2333c5d-9e05-4b18-bfe2-83c5a51d9858 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:52,065 [242] [DEBUG] [app] Starting request: urn:request:57da0d2d-60ba-48b3-a813-60d6dfec813e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:03:52,066 [230] [DEBUG] [app] Ending request: urn:request:c2333c5d-9e05-4b18-bfe2-83c5a51d9858 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2333c5d-9e05-4b18-bfe2-83c5a51d9858', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:03:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:03:52,066 [242] [DEBUG] [app] Ending request: urn:request:57da0d2d-60ba-48b3-a813-60d6dfec813e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57da0d2d-60ba-48b3-a813-60d6dfec813e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:03:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:03:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:03:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:03:52,070 [235] [DEBUG] [app] Starting request: urn:request:7022a1cc-622a-42e8-b3f6-32c012119f57 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:52,070 [236] [DEBUG] [app] Starting request: urn:request:198d3f16-76e4-4623-8aef-6ede840b3f5d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:03:52,070 [235] [DEBUG] [app] Ending request: urn:request:7022a1cc-622a-42e8-b3f6-32c012119f57 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7022a1cc-622a-42e8-b3f6-32c012119f57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:52,071 [236] [DEBUG] [app] Ending request: urn:request:198d3f16-76e4-4623-8aef-6ede840b3f5d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:198d3f16-76e4-4623-8aef-6ede840b3f5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:03:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:03:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:03:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:03:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:03:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:03:52,088 [231] [DEBUG] [app] Ending request: urn:request:0ffb9584-845e-401f-88ca-2e2c13257fd6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ffb9584-845e-401f-88ca-2e2c13257fd6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:03:52,096 [236] [DEBUG] [app] Ending request: urn:request:2d709374-258f-4aaf-81bb-2e3e198ed387 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d709374-258f-4aaf-81bb-2e3e198ed387', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:03:52,096 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:03:52,096 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:03:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.037 47 0.037) notificationworker stdout | 2023-01-17 06:03:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:03:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:01.636362+00:00 (in 5.989553 seconds) notificationworker stdout | 2023-01-17 06:03:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:05 UTC)" (scheduled at 2023-01-17 06:03:55.646354+00:00) notificationworker stdout | 2023-01-17 06:03:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:03:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 3, 55, 647218), True, datetime.datetime(2023, 1, 17, 6, 3, 55, 647218), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:03:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:03:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:03:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:05 UTC)" executed successfully builder stdout | 2023-01-17 06:03:57,305 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:03:57,317 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:03:57,317 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:03:57,328 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:00,495 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:04:00,583 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:04:01,510 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:04:01,512 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:04:01,515 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:04:01,519 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:04:01,521 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:04:01,560 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:04:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:05.646354+00:00 (in 4.009521 seconds) notificationworker stdout | 2023-01-17 06:04:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:01 UTC)" (scheduled at 2023-01-17 06:04:01.636362+00:00) notificationworker stdout | 2023-01-17 06:04:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:04:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:01 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:04:01,945 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:04:02,035 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:04:02,087 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:04:02,582 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:04:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:04:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:19.346120+00:00 (in 14.408818 seconds) repositorygcworker stdout | 2023-01-17 06:04:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:04 UTC)" (scheduled at 2023-01-17 06:04:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:04:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:04:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 4, 937735), True, datetime.datetime(2023, 1, 17, 6, 4, 4, 937735), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:04:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:04:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:04:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:04:05,500 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:04:05,503 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:04:05,505 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:04:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:15.646354+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 06:04:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:15 UTC)" (scheduled at 2023-01-17 06:04:05.646354+00:00) notificationworker stdout | 2023-01-17 06:04:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 5, 647255), True, datetime.datetime(2023, 1, 17, 6, 4, 5, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:15 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:04:05,720 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:04:05,799 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:04:05,962 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:04:05,972 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:04:05,978 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:04:05,981 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:04:05,983 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:04:05,986 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:04:05,989 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:04:05,991 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:04:05,994 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:04:06,840 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:04:06,886 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:04:07,060 [235] [DEBUG] [app] Starting request: urn:request:eef22b81-4d04-4768-8348-d07a66ee3db7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:07,060 [236] [DEBUG] [app] Starting request: urn:request:458ed4d2-d3a4-46a0-93ae-52b6dc295d23 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:07,065 [230] [DEBUG] [app] Starting request: urn:request:a96f048e-d569-41b6-8a88-e2152a458308 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:07,065 [242] [DEBUG] [app] Starting request: urn:request:9912cee1-c38a-423b-bee9-5d2796254546 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:07,065 [230] [DEBUG] [app] Ending request: urn:request:a96f048e-d569-41b6-8a88-e2152a458308 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a96f048e-d569-41b6-8a88-e2152a458308', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:04:07,066 [242] [DEBUG] [app] Ending request: urn:request:9912cee1-c38a-423b-bee9-5d2796254546 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9912cee1-c38a-423b-bee9-5d2796254546', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:04:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:04:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:07,070 [231] [DEBUG] [app] Starting request: urn:request:923f5225-13c1-4f58-9248-59470e07fcac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:07,070 [236] [DEBUG] [app] Starting request: urn:request:5b432f1f-90f7-4b58-ab39-0d50292d394c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:07,070 [231] [DEBUG] [app] Ending request: urn:request:923f5225-13c1-4f58-9248-59470e07fcac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:923f5225-13c1-4f58-9248-59470e07fcac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:07,070 [236] [DEBUG] [app] Ending request: urn:request:5b432f1f-90f7-4b58-ab39-0d50292d394c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5b432f1f-90f7-4b58-ab39-0d50292d394c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:07,087 [236] [DEBUG] [app] Ending request: urn:request:458ed4d2-d3a4-46a0-93ae-52b6dc295d23 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:458ed4d2-d3a4-46a0-93ae-52b6dc295d23', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:07,087 [235] [DEBUG] [app] Ending request: urn:request:eef22b81-4d04-4768-8348-d07a66ee3db7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eef22b81-4d04-4768-8348-d07a66ee3db7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:04:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) repositorygcworker stdout | 2023-01-17 06:04:07,244 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 06:04:07,338 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:07,351 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:07,351 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:07,361 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:04:07,807 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:04:07,920 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:04:08,064 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:04:08,226 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:04:08,783 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:04:09,239 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:04:09,400 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:04:09,663 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:04:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:04:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:40.738568+00:00 (in 29.999512 seconds) buildlogsarchiver stdout | 2023-01-17 06:04:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:40 UTC)" (scheduled at 2023-01-17 06:04:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:04:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 4, 10, 739523), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:04:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:04:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:04:10,752 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:25.646354+00:00 (in 9.999518 seconds) notificationworker stdout | 2023-01-17 06:04:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:25 UTC)" (scheduled at 2023-01-17 06:04:15.646354+00:00) notificationworker stdout | 2023-01-17 06:04:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 15, 647262), True, datetime.datetime(2023, 1, 17, 6, 4, 15, 647262), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:04:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:19.537472+00:00 (in 2.600010 seconds) securityworker stdout | 2023-01-17 06:04:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:46 UTC)" (scheduled at 2023-01-17 06:04:16.936988+00:00) securityworker stdout | 2023-01-17 06:04:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:46 UTC)" executed successfully builder stdout | 2023-01-17 06:04:17,371 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:17,389 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:17,389 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:17,399 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:04:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:04:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:04.936835+00:00 (in 45.590270 seconds) repositorygcworker stdout | 2023-01-17 06:04:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:19 UTC)" (scheduled at 2023-01-17 06:04:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:04:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:04:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:04:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:46.936988+00:00 (in 27.399028 seconds) securityworker stdout | 2023-01-17 06:04:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:49 UTC)" (scheduled at 2023-01-17 06:04:19.537472+00:00) securityworker stdout | 2023-01-17 06:04:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:04:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:04:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:04:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:04:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:04:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:04:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:04:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:04:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:51.348605+00:00 (in 29.999557 seconds) gcworker stdout | 2023-01-17 06:04:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:51 UTC)" (scheduled at 2023-01-17 06:04:21.348605+00:00) gcworker stdout | 2023-01-17 06:04:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725861349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:04:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:04:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:04:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:04:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:04:22,060 [231] [DEBUG] [app] Starting request: urn:request:7a61fe7c-4aca-47ca-8e2e-ed35f0598c5c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:22,060 [233] [DEBUG] [app] Starting request: urn:request:37f6cd38-d075-4de2-bb83-1cc0dbac3fbf (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:22,064 [230] [DEBUG] [app] Starting request: urn:request:0c2f9545-0898-4f35-97b3-69f8e9a83445 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:22,065 [242] [DEBUG] [app] Starting request: urn:request:53a179ed-0e19-4b4c-ac6c-0a4b89f6e583 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:22,065 [230] [DEBUG] [app] Ending request: urn:request:0c2f9545-0898-4f35-97b3-69f8e9a83445 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c2f9545-0898-4f35-97b3-69f8e9a83445', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:04:22,065 [242] [DEBUG] [app] Ending request: urn:request:53a179ed-0e19-4b4c-ac6c-0a4b89f6e583 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:53a179ed-0e19-4b4c-ac6c-0a4b89f6e583', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:04:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:04:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:22,070 [235] [DEBUG] [app] Starting request: urn:request:57511080-e751-4b5f-b2ec-00456066aa83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:22,070 [236] [DEBUG] [app] Starting request: urn:request:50e95a86-20df-4ab8-b40d-ede10d7c9de6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:22,070 [235] [DEBUG] [app] Ending request: urn:request:57511080-e751-4b5f-b2ec-00456066aa83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57511080-e751-4b5f-b2ec-00456066aa83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:22,071 [236] [DEBUG] [app] Ending request: urn:request:50e95a86-20df-4ab8-b40d-ede10d7c9de6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50e95a86-20df-4ab8-b40d-ede10d7c9de6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:04:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:22,087 [233] [DEBUG] [app] Ending request: urn:request:37f6cd38-d075-4de2-bb83-1cc0dbac3fbf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37f6cd38-d075-4de2-bb83-1cc0dbac3fbf', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:22,087 [231] [DEBUG] [app] Ending request: urn:request:7a61fe7c-4aca-47ca-8e2e-ed35f0598c5c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a61fe7c-4aca-47ca-8e2e-ed35f0598c5c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:22,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:04:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) notificationworker stdout | 2023-01-17 06:04:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:35.646354+00:00 (in 9.999519 seconds) notificationworker stdout | 2023-01-17 06:04:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:35 UTC)" (scheduled at 2023-01-17 06:04:25.646354+00:00) notificationworker stdout | 2023-01-17 06:04:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 25, 647265), True, datetime.datetime(2023, 1, 17, 6, 4, 25, 647265), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:35 UTC)" executed successfully builder stdout | 2023-01-17 06:04:27,409 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:27,421 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:27,421 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:27,432 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:04:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:04:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:35.853396+00:00 (in 6.889256 seconds) exportactionlogsworker stdout | 2023-01-17 06:04:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:28 UTC)" (scheduled at 2023-01-17 06:04:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:04:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:04:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:30,531 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:04:30,616 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:04:31,517 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:04:31,519 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:04:31,522 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:04:31,526 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:04:31,528 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:04:31,574 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:04:31,981 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:04:32,070 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:04:32,101 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:04:32,592 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:04:35,507 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:04:35,510 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:04:35,513 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:04:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:45.646354+00:00 (in 9.999557 seconds) notificationworker stdout | 2023-01-17 06:04:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:45 UTC)" (scheduled at 2023-01-17 06:04:35.646354+00:00) notificationworker stdout | 2023-01-17 06:04:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 35, 647216), True, datetime.datetime(2023, 1, 17, 6, 4, 35, 647216), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:45 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:04:35,738 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:04:35,827 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} exportactionlogsworker stdout | 2023-01-17 06:04:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:04:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:28.963662+00:00 (in 53.109751 seconds) exportactionlogsworker stdout | 2023-01-17 06:04:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:35 UTC)" (scheduled at 2023-01-17 06:04:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:04:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:04:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 35, 854299), True, datetime.datetime(2023, 1, 17, 6, 4, 35, 854299), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:04:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:04:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:04:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:04:35,971 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:04:35,980 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:04:35,986 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:04:35,989 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:04:35,991 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:04:35,993 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:04:35,996 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:04:35,999 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:04:36,002 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:04:36,876 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:04:36,914 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:04:37,060 [236] [DEBUG] [app] Starting request: urn:request:5fa6ca9c-3aa8-4571-9c88-9b708e145702 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:37,060 [235] [DEBUG] [app] Starting request: urn:request:d04aca80-78d5-4b33-80f4-260df99db6e9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:37,065 [230] [DEBUG] [app] Starting request: urn:request:9287b018-e45e-49cc-a76b-bbd3a4ca579b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:37,066 [230] [DEBUG] [app] Ending request: urn:request:9287b018-e45e-49cc-a76b-bbd3a4ca579b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9287b018-e45e-49cc-a76b-bbd3a4ca579b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:37,066 [242] [DEBUG] [app] Starting request: urn:request:b0734a22-53be-4ff4-8dac-b256ddb39b38 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:04:37,066 [242] [DEBUG] [app] Ending request: urn:request:b0734a22-53be-4ff4-8dac-b256ddb39b38 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b0734a22-53be-4ff4-8dac-b256ddb39b38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:04:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:04:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:37,070 [231] [DEBUG] [app] Starting request: urn:request:8df81d44-0572-4f95-85ff-d268c276de3e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:37,070 [231] [DEBUG] [app] Ending request: urn:request:8df81d44-0572-4f95-85ff-d268c276de3e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8df81d44-0572-4f95-85ff-d268c276de3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:37,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:37,072 [233] [DEBUG] [app] Starting request: urn:request:f300d7b0-9348-4581-87b9-e11ed4c9c9d3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:37,072 [233] [DEBUG] [app] Ending request: urn:request:f300d7b0-9348-4581-87b9-e11ed4c9c9d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f300d7b0-9348-4581-87b9-e11ed4c9c9d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:37,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:37,087 [235] [DEBUG] [app] Ending request: urn:request:d04aca80-78d5-4b33-80f4-260df99db6e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d04aca80-78d5-4b33-80f4-260df99db6e9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:37,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:04:37,089 [236] [DEBUG] [app] Ending request: urn:request:5fa6ca9c-3aa8-4571-9c88-9b708e145702 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5fa6ca9c-3aa8-4571-9c88-9b708e145702', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:37,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.031) securityscanningnotificationworker stdout | 2023-01-17 06:04:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:04:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:37.537407+00:00 (in 0.280327 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:04:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:37 UTC)" (scheduled at 2023-01-17 06:04:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:04:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:04:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 37, 257472), True, datetime.datetime(2023, 1, 17, 6, 4, 37, 257472), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:04:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:04:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:04:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:04:37,279 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 06:04:37,442 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:37,454 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:37,454 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:37,464 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:04:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:04:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:37.256652+00:00 (in 59.718805 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:04:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:37 UTC)" (scheduled at 2023-01-17 06:04:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:04:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:04:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 06:04:37,841 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:04:37,927 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:04:38,075 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:04:38,249 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:04:38,788 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:04:39,275 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:04:39,436 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:04:39,668 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:04:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:04:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:10.738568+00:00 (in 29.999453 seconds) buildlogsarchiver stdout | 2023-01-17 06:04:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:10 UTC)" (scheduled at 2023-01-17 06:04:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:04:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 4, 40, 739587), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:04:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:04:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:04:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:04:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:04:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:42.941536+00:00 (in 1.792798 seconds) namespacegcworker stdout | 2023-01-17 06:04:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:41 UTC)" (scheduled at 2023-01-17 06:04:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:04:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:04:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 41, 149152), True, datetime.datetime(2023, 1, 17, 6, 4, 41, 149152), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:04:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:04:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:04:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:04:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:04:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:41.148245+00:00 (in 58.206220 seconds) namespacegcworker stdout | 2023-01-17 06:04:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:42 UTC)" (scheduled at 2023-01-17 06:04:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:04:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:04:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:05:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:04:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:55.646354+00:00 (in 9.999545 seconds) notificationworker stdout | 2023-01-17 06:04:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:55 UTC)" (scheduled at 2023-01-17 06:04:45.646354+00:00) notificationworker stdout | 2023-01-17 06:04:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 45, 647229), True, datetime.datetime(2023, 1, 17, 6, 4, 45, 647229), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:04:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:04:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:04:49.537472+00:00 (in 2.600027 seconds) securityworker stdout | 2023-01-17 06:04:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:16 UTC)" (scheduled at 2023-01-17 06:04:46.936988+00:00) securityworker stdout | 2023-01-17 06:04:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:16 UTC)" executed successfully builder stdout | 2023-01-17 06:04:47,474 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:47,487 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:47,487 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:47,498 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:04:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:16.936988+00:00 (in 27.399051 seconds) securityworker stdout | 2023-01-17 06:04:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:19 UTC)" (scheduled at 2023-01-17 06:04:49.537472+00:00) securityworker stdout | 2023-01-17 06:04:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:04:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:04:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:49,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:04:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:04:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:04:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:04:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:04:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:04:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:04:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:04:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:04:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:04:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:04:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:04:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:04:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:21.348605+00:00 (in 29.999541 seconds) gcworker stdout | 2023-01-17 06:04:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:21 UTC)" (scheduled at 2023-01-17 06:04:51.348605+00:00) gcworker stdout | 2023-01-17 06:04:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725891349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:04:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:04:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:04:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:04:52,059 [236] [DEBUG] [app] Starting request: urn:request:829b0d4f-0f7e-43de-b444-39fda3a28bc6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:52,059 [233] [DEBUG] [app] Starting request: urn:request:f7713da3-1674-473c-a6f2-8fc85589f807 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:04:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:04:52,064 [242] [DEBUG] [app] Starting request: urn:request:8a88c46c-b3d1-4ce9-bd4f-f1e16ebdfb00 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:52,065 [243] [DEBUG] [app] Starting request: urn:request:55b4cfb8-54b8-4bbf-b15e-f972aae0323e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:04:52,065 [242] [DEBUG] [app] Ending request: urn:request:8a88c46c-b3d1-4ce9-bd4f-f1e16ebdfb00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a88c46c-b3d1-4ce9-bd4f-f1e16ebdfb00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:04:52,065 [243] [DEBUG] [app] Ending request: urn:request:55b4cfb8-54b8-4bbf-b15e-f972aae0323e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:55b4cfb8-54b8-4bbf-b15e-f972aae0323e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:04:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:04:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:04:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:04:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:04:52,070 [231] [DEBUG] [app] Starting request: urn:request:c36367ac-a371-4e04-bee3-0b1b45a58227 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:52,070 [231] [DEBUG] [app] Ending request: urn:request:c36367ac-a371-4e04-bee3-0b1b45a58227 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c36367ac-a371-4e04-bee3-0b1b45a58227', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:52,070 [235] [DEBUG] [app] Starting request: urn:request:d3dae548-0573-4ed4-8a78-11a71feb1a5e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:04:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:04:52,071 [235] [DEBUG] [app] Ending request: urn:request:d3dae548-0573-4ed4-8a78-11a71feb1a5e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3dae548-0573-4ed4-8a78-11a71feb1a5e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:04:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:04:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:04:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:04:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:04:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:04:52,087 [236] [DEBUG] [app] Ending request: urn:request:829b0d4f-0f7e-43de-b444-39fda3a28bc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:829b0d4f-0f7e-43de-b444-39fda3a28bc6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:52,087 [233] [DEBUG] [app] Ending request: urn:request:f7713da3-1674-473c-a6f2-8fc85589f807 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f7713da3-1674-473c-a6f2-8fc85589f807', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:04:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:52,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:04:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:04:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:04:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:04:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:04:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:01.636362+00:00 (in 5.989520 seconds) notificationworker stdout | 2023-01-17 06:04:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:05 UTC)" (scheduled at 2023-01-17 06:04:55.646354+00:00) notificationworker stdout | 2023-01-17 06:04:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:04:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 4, 55, 647240), True, datetime.datetime(2023, 1, 17, 6, 4, 55, 647240), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:04:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:04:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:04:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:05 UTC)" executed successfully builder stdout | 2023-01-17 06:04:57,507 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:04:57,519 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:04:57,519 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:04:57,530 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:00,566 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:05:00,638 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:05:01,524 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:05:01,527 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:05:01,529 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:05:01,532 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:05:01,534 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:05:01,600 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} notificationworker stdout | 2023-01-17 06:05:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:05.646354+00:00 (in 4.009552 seconds) notificationworker stdout | 2023-01-17 06:05:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:01 UTC)" (scheduled at 2023-01-17 06:05:01.636362+00:00) notificationworker stdout | 2023-01-17 06:05:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:05:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:01 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:05:02,016 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:05:02,106 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:05:02,108 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:05:02,622 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:05:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:05:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:19.346120+00:00 (in 14.408842 seconds) repositorygcworker stdout | 2023-01-17 06:05:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:04 UTC)" (scheduled at 2023-01-17 06:05:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:05:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:05:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 4, 937724), True, datetime.datetime(2023, 1, 17, 6, 5, 4, 937724), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:05:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:05:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:05:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:05:05,515 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:05:05,518 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:05:05,520 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:05:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:15.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 06:05:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:15 UTC)" (scheduled at 2023-01-17 06:05:05.646354+00:00) notificationworker stdout | 2023-01-17 06:05:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 5, 647211), True, datetime.datetime(2023, 1, 17, 6, 5, 5, 647211), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:15 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:05:05,774 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:05:05,862 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:05:05,981 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:05:05,987 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:05:05,994 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:05:05,996 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:05:05,998 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:05:06,001 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:05:06,004 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:05:06,006 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:05:06,008 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:05:06,911 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:05:06,940 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:05:07,060 [231] [DEBUG] [app] Starting request: urn:request:d4598293-523f-48ca-9398-96a477c0fa3b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:07,060 [236] [DEBUG] [app] Starting request: urn:request:d014d104-0f09-41c5-b3c0-0cafe9b77a3d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:07,065 [230] [DEBUG] [app] Starting request: urn:request:792cea10-5180-40bd-a7ee-20085d2b5cd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:07,066 [230] [DEBUG] [app] Ending request: urn:request:792cea10-5180-40bd-a7ee-20085d2b5cd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:792cea10-5180-40bd-a7ee-20085d2b5cd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:07,066 [243] [DEBUG] [app] Starting request: urn:request:1e04c812-d1bb-4840-a354-65c9348fa130 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:05:07,066 [243] [DEBUG] [app] Ending request: urn:request:1e04c812-d1bb-4840-a354-65c9348fa130 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e04c812-d1bb-4840-a354-65c9348fa130', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:05:07,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:05:07,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,069 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:07,070 [233] [DEBUG] [app] Starting request: urn:request:b4a6620e-92fd-44db-a12b-e4c73cc9cf10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:07,070 [233] [DEBUG] [app] Ending request: urn:request:b4a6620e-92fd-44db-a12b-e4c73cc9cf10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4a6620e-92fd-44db-a12b-e4c73cc9cf10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:07,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:07,072 [235] [DEBUG] [app] Starting request: urn:request:8738d15e-02b8-494b-bf9e-479963f19ddb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:07,072 [235] [DEBUG] [app] Ending request: urn:request:8738d15e-02b8-494b-bf9e-479963f19ddb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8738d15e-02b8-494b-bf9e-479963f19ddb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:07,073 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:07,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:07,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:07,087 [231] [DEBUG] [app] Ending request: urn:request:d4598293-523f-48ca-9398-96a477c0fa3b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d4598293-523f-48ca-9398-96a477c0fa3b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:05:07,089 [236] [DEBUG] [app] Ending request: urn:request:d014d104-0f09-41c5-b3c0-0cafe9b77a3d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d014d104-0f09-41c5-b3c0-0cafe9b77a3d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:07,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) repositorygcworker stdout | 2023-01-17 06:05:07,315 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 06:05:07,540 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:07,553 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:07,553 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:07,563 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:05:07,847 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:05:07,953 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:05:08,101 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:05:08,283 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:05:08,824 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:05:09,311 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:05:09,462 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:05:09,687 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:05:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:05:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:40.738568+00:00 (in 29.999423 seconds) buildlogsarchiver stdout | 2023-01-17 06:05:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:40 UTC)" (scheduled at 2023-01-17 06:05:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:05:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 5, 10, 739606), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:05:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:05:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:05:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:05:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:25.646354+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 06:05:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:25 UTC)" (scheduled at 2023-01-17 06:05:15.646354+00:00) notificationworker stdout | 2023-01-17 06:05:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 15, 647184), True, datetime.datetime(2023, 1, 17, 6, 5, 15, 647184), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:05:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:19.537472+00:00 (in 2.600008 seconds) securityworker stdout | 2023-01-17 06:05:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:46 UTC)" (scheduled at 2023-01-17 06:05:16.936988+00:00) securityworker stdout | 2023-01-17 06:05:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:46 UTC)" executed successfully builder stdout | 2023-01-17 06:05:17,573 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:17,586 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:17,586 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:17,596 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:05:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:05:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:04.936835+00:00 (in 45.590251 seconds) repositorygcworker stdout | 2023-01-17 06:05:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:19 UTC)" (scheduled at 2023-01-17 06:05:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:05:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:05:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:05:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:46.936988+00:00 (in 27.399094 seconds) securityworker stdout | 2023-01-17 06:05:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:49 UTC)" (scheduled at 2023-01-17 06:05:19.537472+00:00) securityworker stdout | 2023-01-17 06:05:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:05:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:05:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:05:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:05:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:05:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:05:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:05:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:05:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:51.348605+00:00 (in 29.999509 seconds) gcworker stdout | 2023-01-17 06:05:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:51 UTC)" (scheduled at 2023-01-17 06:05:21.348605+00:00) gcworker stdout | 2023-01-17 06:05:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725921349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:05:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:05:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:05:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:05:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:05:22,060 [231] [DEBUG] [app] Starting request: urn:request:6a4580c3-bc54-4921-a095-9d0f1c47d956 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:22,060 [236] [DEBUG] [app] Starting request: urn:request:8bc8ca53-894c-4788-8f98-88377e35014a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:22,064 [243] [DEBUG] [app] Starting request: urn:request:228fa34f-e898-4736-8b3d-30ada4a28a35 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:22,065 [242] [DEBUG] [app] Starting request: urn:request:11ecfe28-7491-4a62-9d99-6055aea77f09 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:22,065 [243] [DEBUG] [app] Ending request: urn:request:228fa34f-e898-4736-8b3d-30ada4a28a35 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:228fa34f-e898-4736-8b3d-30ada4a28a35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:22,065 [242] [DEBUG] [app] Ending request: urn:request:11ecfe28-7491-4a62-9d99-6055aea77f09 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11ecfe28-7491-4a62-9d99-6055aea77f09', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:05:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:05:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:22,069 [233] [DEBUG] [app] Starting request: urn:request:d5dc2678-b778-42b6-a21c-9e9016ff935c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:22,070 [235] [DEBUG] [app] Starting request: urn:request:1ee4a698-5c2c-442e-a5b9-944ee0c26bee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:22,070 [233] [DEBUG] [app] Ending request: urn:request:d5dc2678-b778-42b6-a21c-9e9016ff935c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5dc2678-b778-42b6-a21c-9e9016ff935c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:05:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:22,070 [235] [DEBUG] [app] Ending request: urn:request:1ee4a698-5c2c-442e-a5b9-944ee0c26bee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1ee4a698-5c2c-442e-a5b9-944ee0c26bee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:22,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:22,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:22,087 [236] [DEBUG] [app] Ending request: urn:request:8bc8ca53-894c-4788-8f98-88377e35014a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8bc8ca53-894c-4788-8f98-88377e35014a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:22,087 [231] [DEBUG] [app] Ending request: urn:request:6a4580c3-bc54-4921-a095-9d0f1c47d956 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a4580c3-bc54-4921-a095-9d0f1c47d956', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:22,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:05:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) notificationworker stdout | 2023-01-17 06:05:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:35.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 06:05:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:35 UTC)" (scheduled at 2023-01-17 06:05:25.646354+00:00) notificationworker stdout | 2023-01-17 06:05:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 25, 647199), True, datetime.datetime(2023, 1, 17, 6, 5, 25, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:35 UTC)" executed successfully builder stdout | 2023-01-17 06:05:27,607 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:27,619 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:27,619 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:27,630 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:05:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:05:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:35.853396+00:00 (in 6.889301 seconds) exportactionlogsworker stdout | 2023-01-17 06:05:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:28 UTC)" (scheduled at 2023-01-17 06:05:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:05:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:05:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:05:30,579 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:05:30,674 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:05:31,531 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:05:31,534 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:05:31,536 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:05:31,539 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:05:31,541 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:05:31,634 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:05:32,049 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:05:32,121 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:05:32,141 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:05:32,642 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:05:35,523 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:05:35,525 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:05:35,527 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:05:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:45.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 06:05:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:45 UTC)" (scheduled at 2023-01-17 06:05:35.646354+00:00) notificationworker stdout | 2023-01-17 06:05:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 35, 647216), True, datetime.datetime(2023, 1, 17, 6, 5, 35, 647216), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:45 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:05:35,809 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} exportactionlogsworker stdout | 2023-01-17 06:05:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:05:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:28.963662+00:00 (in 53.109855 seconds) exportactionlogsworker stdout | 2023-01-17 06:05:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:35 UTC)" (scheduled at 2023-01-17 06:05:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:05:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:05:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 35, 854181), True, datetime.datetime(2023, 1, 17, 6, 5, 35, 854181), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:05:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:05:35,864 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:05:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:35 UTC)" executed successfully expiredappspecifictokenworker stdout | 2023-01-17 06:05:35,866 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:05:35,991 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:05:35,993 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:05:36,000 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:05:36,003 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:05:36,007 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:05:36,009 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:05:36,013 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:05:36,017 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:05:36,019 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:05:36,941 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:05:36,971 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:05:37,060 [236] [DEBUG] [app] Starting request: urn:request:92b1d632-a1c8-43cf-a7cb-1dcd59f84172 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:37,060 [235] [DEBUG] [app] Starting request: urn:request:e03fe14f-b4b9-48dc-9b35-b689a0b48d22 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:37,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:37,065 [243] [DEBUG] [app] Starting request: urn:request:a331d079-0275-4a58-bb4f-2f97dde8986a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:37,065 [242] [DEBUG] [app] Starting request: urn:request:d1d56b4b-94d2-403b-b5b2-50c1792948d0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:37,066 [243] [DEBUG] [app] Ending request: urn:request:a331d079-0275-4a58-bb4f-2f97dde8986a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a331d079-0275-4a58-bb4f-2f97dde8986a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:37,066 [242] [DEBUG] [app] Ending request: urn:request:d1d56b4b-94d2-403b-b5b2-50c1792948d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d1d56b4b-94d2-403b-b5b2-50c1792948d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:05:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:05:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:05:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:37,071 [231] [DEBUG] [app] Starting request: urn:request:82338f31-81f1-47ff-bebc-aca51e0438e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:37,071 [233] [DEBUG] [app] Starting request: urn:request:fbfbd84c-b8e7-4777-bb8f-f78a7b37cf51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:37,071 [231] [DEBUG] [app] Ending request: urn:request:82338f31-81f1-47ff-bebc-aca51e0438e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82338f31-81f1-47ff-bebc-aca51e0438e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:05:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:37,071 [233] [DEBUG] [app] Ending request: urn:request:fbfbd84c-b8e7-4777-bb8f-f78a7b37cf51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fbfbd84c-b8e7-4777-bb8f-f78a7b37cf51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:37,088 [235] [DEBUG] [app] Ending request: urn:request:e03fe14f-b4b9-48dc-9b35-b689a0b48d22 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e03fe14f-b4b9-48dc-9b35-b689a0b48d22', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:37,088 [236] [DEBUG] [app] Ending request: urn:request:92b1d632-a1c8-43cf-a7cb-1dcd59f84172 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:92b1d632-a1c8-43cf-a7cb-1dcd59f84172', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:05:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:05:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:05:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:37.537407+00:00 (in 0.280356 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:05:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:37 UTC)" (scheduled at 2023-01-17 06:05:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:05:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:05:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 37, 257484), True, datetime.datetime(2023, 1, 17, 6, 5, 37, 257484), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:05:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:05:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:05:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:05:37,339 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} securityscanningnotificationworker stdout | 2023-01-17 06:05:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:05:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:23.537048+00:00 (in 45.999068 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:05:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:37 UTC)" (scheduled at 2023-01-17 06:05:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:05:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:05:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:37 UTC)" executed successfully builder stdout | 2023-01-17 06:05:37,640 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:37,652 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:37,652 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:37,662 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:05:37,880 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:05:37,958 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:05:38,108 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:05:38,318 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:05:38,830 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:05:39,336 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:05:39,475 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:05:39,696 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:05:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:05:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:10.738568+00:00 (in 29.999466 seconds) buildlogsarchiver stdout | 2023-01-17 06:05:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:10 UTC)" (scheduled at 2023-01-17 06:05:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:05:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 5, 40, 739483), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:05:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:05:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:05:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:05:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:05:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:42.941536+00:00 (in 1.792799 seconds) namespacegcworker stdout | 2023-01-17 06:05:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:41 UTC)" (scheduled at 2023-01-17 06:05:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:05:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:05:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 41, 149152), True, datetime.datetime(2023, 1, 17, 6, 5, 41, 149152), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:05:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:05:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:05:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:05:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:05:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:41.148245+00:00 (in 58.206244 seconds) namespacegcworker stdout | 2023-01-17 06:05:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:42 UTC)" (scheduled at 2023-01-17 06:05:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:05:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:05:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:06:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:05:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:55.646354+00:00 (in 9.999589 seconds) notificationworker stdout | 2023-01-17 06:05:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:55 UTC)" (scheduled at 2023-01-17 06:05:45.646354+00:00) notificationworker stdout | 2023-01-17 06:05:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 45, 647192), True, datetime.datetime(2023, 1, 17, 6, 5, 45, 647192), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:05:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:05:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:05:49.537472+00:00 (in 2.600053 seconds) securityworker stdout | 2023-01-17 06:05:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:16 UTC)" (scheduled at 2023-01-17 06:05:46.936988+00:00) securityworker stdout | 2023-01-17 06:05:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:16 UTC)" executed successfully builder stdout | 2023-01-17 06:05:47,668 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:47,681 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:47,681 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:47,691 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:05:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:16.936988+00:00 (in 27.399044 seconds) securityworker stdout | 2023-01-17 06:05:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:19 UTC)" (scheduled at 2023-01-17 06:05:49.537472+00:00) securityworker stdout | 2023-01-17 06:05:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:05:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:05:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:05:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:05:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:05:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:05:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:05:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:05:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:05:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:05:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:05:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:05:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:05:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:05:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:05:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:21.348605+00:00 (in 29.999452 seconds) gcworker stdout | 2023-01-17 06:05:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:21 UTC)" (scheduled at 2023-01-17 06:05:51.348605+00:00) gcworker stdout | 2023-01-17 06:05:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725951349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:05:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:05:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:05:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:05:52,059 [236] [DEBUG] [app] Starting request: urn:request:2618ef4a-cda1-4a00-bdb6-5d517f5eab46 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:52,059 [231] [DEBUG] [app] Starting request: urn:request:b2242175-abf1-46d5-ba97-8fe5375ad22e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:05:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:05:52,064 [230] [DEBUG] [app] Starting request: urn:request:d025a4f5-33a1-4ac8-a5fb-d3edc5218eec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:52,065 [241] [DEBUG] [app] Starting request: urn:request:9e793a34-f1c3-4ffd-bad3-a6266bdab4a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:05:52,065 [230] [DEBUG] [app] Ending request: urn:request:d025a4f5-33a1-4ac8-a5fb-d3edc5218eec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d025a4f5-33a1-4ac8-a5fb-d3edc5218eec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:05:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:05:52,065 [241] [DEBUG] [app] Ending request: urn:request:9e793a34-f1c3-4ffd-bad3-a6266bdab4a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e793a34-f1c3-4ffd-bad3-a6266bdab4a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:05:52,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:05:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:52,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:05:52,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:05:52,069 [235] [DEBUG] [app] Starting request: urn:request:139d0cad-d60e-4fd7-aef2-26d186a9eda4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:52,069 [231] [DEBUG] [app] Starting request: urn:request:f92ecd8b-b9ae-44c4-966f-a7d86fd3ca8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:05:52,070 [235] [DEBUG] [app] Ending request: urn:request:139d0cad-d60e-4fd7-aef2-26d186a9eda4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:139d0cad-d60e-4fd7-aef2-26d186a9eda4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:05:52,070 [231] [DEBUG] [app] Ending request: urn:request:f92ecd8b-b9ae-44c4-966f-a7d86fd3ca8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f92ecd8b-b9ae-44c4-966f-a7d86fd3ca8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:05:52,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:05:52,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:05:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:05:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:52,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:05:52,086 [231] [DEBUG] [app] Ending request: urn:request:b2242175-abf1-46d5-ba97-8fe5375ad22e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2242175-abf1-46d5-ba97-8fe5375ad22e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:05:52,087 [236] [DEBUG] [app] Ending request: urn:request:2618ef4a-cda1-4a00-bdb6-5d517f5eab46 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2618ef4a-cda1-4a00-bdb6-5d517f5eab46', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:05:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:05:52,087 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:05:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:05:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:05:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:05:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:01.636362+00:00 (in 5.989524 seconds) notificationworker stdout | 2023-01-17 06:05:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:05 UTC)" (scheduled at 2023-01-17 06:05:55.646354+00:00) notificationworker stdout | 2023-01-17 06:05:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:05:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 5, 55, 647265), True, datetime.datetime(2023, 1, 17, 6, 5, 55, 647265), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:05:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:05:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:05:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:05 UTC)" executed successfully builder stdout | 2023-01-17 06:05:57,701 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:05:57,714 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:05:57,714 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:05:57,725 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:00,591 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:06:00,684 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:06:01,539 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:06:01,542 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:06:01,544 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:06:01,547 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:06:01,550 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:06:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:05.646354+00:00 (in 4.009507 seconds) notificationworker stdout | 2023-01-17 06:06:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:01 UTC)" (scheduled at 2023-01-17 06:06:01.636362+00:00) notificationworker stdout | 2023-01-17 06:06:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:06:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:01 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:06:01,670 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:06:02,056 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:06:02,137 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:06:02,154 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:06:02,647 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:06:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:06:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:17.345601+00:00 (in 12.408327 seconds) repositorygcworker stdout | 2023-01-17 06:06:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:04 UTC)" (scheduled at 2023-01-17 06:06:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:06:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:06:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 4, 937696), True, datetime.datetime(2023, 1, 17, 6, 6, 4, 937696), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:06:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:06:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:06:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:06:05,530 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:06:05,534 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:06:05,536 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:06:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:15.646354+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 06:06:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:15 UTC)" (scheduled at 2023-01-17 06:06:05.646354+00:00) notificationworker stdout | 2023-01-17 06:06:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 5, 647182), True, datetime.datetime(2023, 1, 17, 6, 6, 5, 647182), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:15 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:06:05,831 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:06:05,900 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:06:05,999 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:06:06,002 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:06:06,006 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:06:06,012 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:06:06,017 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:06:06,020 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:06:06,023 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:06:06,026 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:06:06,029 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:06:06,974 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:06:06,999 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:06:07,059 [231] [DEBUG] [app] Starting request: urn:request:15061d67-e309-4ad6-a34b-904a7bc76bc1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:07,060 [236] [DEBUG] [app] Starting request: urn:request:6a8a912e-4dfa-4f47-b644-37b0d3cbccf4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:07,065 [241] [DEBUG] [app] Starting request: urn:request:23624912-3307-4b74-807a-e464ce3f96d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:07,065 [230] [DEBUG] [app] Starting request: urn:request:44690c61-52e5-4581-b902-a370b11c43d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:07,065 [241] [DEBUG] [app] Ending request: urn:request:23624912-3307-4b74-807a-e464ce3f96d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23624912-3307-4b74-807a-e464ce3f96d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:06:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:06:07,066 [230] [DEBUG] [app] Ending request: urn:request:44690c61-52e5-4581-b902-a370b11c43d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44690c61-52e5-4581-b902-a370b11c43d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:06:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:06:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:07,070 [233] [DEBUG] [app] Starting request: urn:request:f6adb0e0-de17-4106-bdf0-299f5c10554f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:07,070 [233] [DEBUG] [app] Ending request: urn:request:f6adb0e0-de17-4106-bdf0-299f5c10554f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6adb0e0-de17-4106-bdf0-299f5c10554f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:07,070 [235] [DEBUG] [app] Starting request: urn:request:f14ca4ff-63d5-44d6-9334-ca6c5366b071 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:07,071 [235] [DEBUG] [app] Ending request: urn:request:f14ca4ff-63d5-44d6-9334-ca6c5366b071 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f14ca4ff-63d5-44d6-9334-ca6c5366b071', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:06:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:07,087 [231] [DEBUG] [app] Ending request: urn:request:15061d67-e309-4ad6-a34b-904a7bc76bc1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15061d67-e309-4ad6-a34b-904a7bc76bc1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:07,087 [236] [DEBUG] [app] Ending request: urn:request:6a8a912e-4dfa-4f47-b644-37b0d3cbccf4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a8a912e-4dfa-4f47-b644-37b0d3cbccf4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:06:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) repositorygcworker stdout | 2023-01-17 06:06:07,356 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 06:06:07,725 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:07,738 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:07,738 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:07,748 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:06:07,903 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:06:07,993 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:06:08,118 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:06:08,342 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:06:08,840 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:06:09,344 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:06:09,484 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:06:09,732 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:06:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:06:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:40.738568+00:00 (in 29.999546 seconds) buildlogsarchiver stdout | 2023-01-17 06:06:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:40 UTC)" (scheduled at 2023-01-17 06:06:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:06:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 6, 10, 739409), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:06:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:06:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:06:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:25.646354+00:00 (in 9.999564 seconds) notificationworker stdout | 2023-01-17 06:06:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:25 UTC)" (scheduled at 2023-01-17 06:06:15.646354+00:00) notificationworker stdout | 2023-01-17 06:06:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 15, 647182), True, datetime.datetime(2023, 1, 17, 6, 6, 15, 647182), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:06:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:19.537472+00:00 (in 2.599981 seconds) securityworker stdout | 2023-01-17 06:06:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:46 UTC)" (scheduled at 2023-01-17 06:06:16.936988+00:00) securityworker stdout | 2023-01-17 06:06:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:06:17,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:06:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:19.346120+00:00 (in 2.000002 seconds) repositorygcworker stdout | 2023-01-17 06:06:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:17 UTC)" (scheduled at 2023-01-17 06:06:17.345601+00:00) repositorygcworker stdout | 2023-01-17 06:06:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 6, 17, 346563), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:06:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 17, 346563), True, datetime.datetime(2023, 1, 17, 6, 6, 17, 346563), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:06:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 17, 346563), True, datetime.datetime(2023, 1, 17, 6, 6, 17, 346563), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 6, 17, 346563), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:06:17,365 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:06:17,365 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:17 UTC)" executed successfully builder stdout | 2023-01-17 06:06:17,758 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:17,771 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:17,771 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:17,781 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:06:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:06:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:04.936835+00:00 (in 45.590225 seconds) repositorygcworker stdout | 2023-01-17 06:06:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:19 UTC)" (scheduled at 2023-01-17 06:06:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:06:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:06:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:06:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:46.936988+00:00 (in 27.399022 seconds) securityworker stdout | 2023-01-17 06:06:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:49 UTC)" (scheduled at 2023-01-17 06:06:19.537472+00:00) securityworker stdout | 2023-01-17 06:06:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:06:19,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:06:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:19,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:19,556 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:06:19,556 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:06:19,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:06:19,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:06:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:06:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:06:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:51.348605+00:00 (in 29.999537 seconds) gcworker stdout | 2023-01-17 06:06:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:51 UTC)" (scheduled at 2023-01-17 06:06:21.348605+00:00) gcworker stdout | 2023-01-17 06:06:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672725981349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:06:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:06:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:06:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:06:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:06:22,060 [231] [DEBUG] [app] Starting request: urn:request:50441874-633e-477a-b984-ff709da61582 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:22,060 [236] [DEBUG] [app] Starting request: urn:request:09f32e99-628e-4581-a94c-61371bd835fb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:22,066 [243] [DEBUG] [app] Starting request: urn:request:1317275a-55e5-4574-9787-7e36eae9afa4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:22,066 [241] [DEBUG] [app] Starting request: urn:request:c05a73c9-9dcb-4974-87d6-deb116cfa7fe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:22,066 [243] [DEBUG] [app] Ending request: urn:request:1317275a-55e5-4574-9787-7e36eae9afa4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1317275a-55e5-4574-9787-7e36eae9afa4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:06:22,066 [241] [DEBUG] [app] Ending request: urn:request:c05a73c9-9dcb-4974-87d6-deb116cfa7fe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c05a73c9-9dcb-4974-87d6-deb116cfa7fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:06:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:06:22,067 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:06:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:22,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:22,071 [233] [DEBUG] [app] Starting request: urn:request:1943f6d9-6398-4cd6-9a72-e9fba7d0be11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:22,072 [235] [DEBUG] [app] Starting request: urn:request:a69fce66-44bf-49cf-986f-f3f55c89cc38 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:22,072 [233] [DEBUG] [app] Ending request: urn:request:1943f6d9-6398-4cd6-9a72-e9fba7d0be11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1943f6d9-6398-4cd6-9a72-e9fba7d0be11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:06:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:22,072 [235] [DEBUG] [app] Ending request: urn:request:a69fce66-44bf-49cf-986f-f3f55c89cc38 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a69fce66-44bf-49cf-986f-f3f55c89cc38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:22,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:22,073 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:22,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:22,080 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:22,089 [231] [DEBUG] [app] Ending request: urn:request:50441874-633e-477a-b984-ff709da61582 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:50441874-633e-477a-b984-ff709da61582', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:22,089 [236] [DEBUG] [app] Ending request: urn:request:09f32e99-628e-4581-a94c-61371bd835fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09f32e99-628e-4581-a94c-61371bd835fb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:22,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:22,089 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 06:06:22,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:06:22,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:06:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:06:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:37.256652+00:00 (in 13.719134 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:06:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:23 UTC)" (scheduled at 2023-01-17 06:06:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:06:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 6, 23, 537847), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:06:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 23, 537847), True, datetime.datetime(2023, 1, 17, 6, 6, 23, 537847), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:06:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 23, 537847), True, datetime.datetime(2023, 1, 17, 6, 6, 23, 537847), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 6, 23, 537847), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:06:23,555 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:06:23,555 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:35.646354+00:00 (in 9.999587 seconds) notificationworker stdout | 2023-01-17 06:06:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:35 UTC)" (scheduled at 2023-01-17 06:06:25.646354+00:00) notificationworker stdout | 2023-01-17 06:06:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 25, 647189), True, datetime.datetime(2023, 1, 17, 6, 6, 25, 647189), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:35 UTC)" executed successfully builder stdout | 2023-01-17 06:06:27,787 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:27,799 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:27,799 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:27,810 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:06:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:06:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:35.853396+00:00 (in 6.889330 seconds) exportactionlogsworker stdout | 2023-01-17 06:06:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:28 UTC)" (scheduled at 2023-01-17 06:06:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:06:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:06:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:30,622 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:06:30,716 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:06:31,548 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:06:31,550 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:06:31,553 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:06:31,555 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:06:31,558 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:06:31,706 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:06:32,091 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:06:32,149 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:06:32,189 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:06:32,653 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:06:35,538 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:06:35,541 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:06:35,543 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:06:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:45.646354+00:00 (in 9.999579 seconds) notificationworker stdout | 2023-01-17 06:06:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:45 UTC)" (scheduled at 2023-01-17 06:06:35.646354+00:00) notificationworker stdout | 2023-01-17 06:06:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 35, 647203), True, datetime.datetime(2023, 1, 17, 6, 6, 35, 647203), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:06:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:06:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:54.963413+00:00 (in 19.109550 seconds) exportactionlogsworker stdout | 2023-01-17 06:06:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:35 UTC)" (scheduled at 2023-01-17 06:06:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:06:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:06:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 35, 854243), True, datetime.datetime(2023, 1, 17, 6, 6, 35, 854243), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:06:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:06:35,864 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:06:35,864 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:35 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:06:35,866 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:06:35,929 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:06:36,007 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:06:36,010 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:06:36,017 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:06:36,022 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:06:36,026 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:06:36,029 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:06:36,031 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:06:36,035 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:06:36,038 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} storagereplication stdout | 2023-01-17 06:06:37,010 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:06:37,025 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:06:37,059 [236] [DEBUG] [app] Starting request: urn:request:a5b579a6-5612-4e4d-9e2a-971abaee05cd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:37,059 [235] [DEBUG] [app] Starting request: urn:request:e17e9a04-f868-442f-aa23-41254a7bd956 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:37,064 [241] [DEBUG] [app] Starting request: urn:request:ff0a2547-a815-4813-bb73-25c2bb92bf34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:37,064 [241] [DEBUG] [app] Ending request: urn:request:ff0a2547-a815-4813-bb73-25c2bb92bf34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff0a2547-a815-4813-bb73-25c2bb92bf34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:06:37,064 [230] [DEBUG] [app] Starting request: urn:request:3cee1700-9840-46cd-ace9-db5b11cf928c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:37,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:06:37,065 [230] [DEBUG] [app] Ending request: urn:request:3cee1700-9840-46cd-ace9-db5b11cf928c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3cee1700-9840-46cd-ace9-db5b11cf928c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:37,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:06:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:06:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:37,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:37,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:37,070 [236] [DEBUG] [app] Starting request: urn:request:5ce5f80a-5523-4434-a740-3a69bb0e80a1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:37,070 [236] [DEBUG] [app] Ending request: urn:request:5ce5f80a-5523-4434-a740-3a69bb0e80a1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ce5f80a-5523-4434-a740-3a69bb0e80a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:37,070 [233] [DEBUG] [app] Starting request: urn:request:e1d42528-8fb7-40e6-8178-8404ce484980 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:37,071 [233] [DEBUG] [app] Ending request: urn:request:e1d42528-8fb7-40e6-8178-8404ce484980 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e1d42528-8fb7-40e6-8178-8404ce484980', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:06:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:37,087 [236] [DEBUG] [app] Ending request: urn:request:a5b579a6-5612-4e4d-9e2a-971abaee05cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5b579a6-5612-4e4d-9e2a-971abaee05cd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:37,088 [235] [DEBUG] [app] Ending request: urn:request:e17e9a04-f868-442f-aa23-41254a7bd956 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e17e9a04-f868-442f-aa23-41254a7bd956', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:06:37,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 06:06:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:06:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:06:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:37.537407+00:00 (in 0.280300 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:06:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:37 UTC)" (scheduled at 2023-01-17 06:06:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:06:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:06:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 37, 257493), True, datetime.datetime(2023, 1, 17, 6, 6, 37, 257493), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:06:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:06:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:06:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:06:37,392 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} securityscanningnotificationworker stdout | 2023-01-17 06:06:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:06:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:37.256652+00:00 (in 59.718825 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:06:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:37 UTC)" (scheduled at 2023-01-17 06:06:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:06:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:06:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:37 UTC)" executed successfully builder stdout | 2023-01-17 06:06:37,814 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:37,826 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:37,826 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:37,837 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:06:37,929 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:06:38,015 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:06:38,124 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:06:38,354 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:06:38,876 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:06:39,380 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:06:39,519 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:06:39,767 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:06:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:06:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:10.738568+00:00 (in 29.999561 seconds) buildlogsarchiver stdout | 2023-01-17 06:06:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:10 UTC)" (scheduled at 2023-01-17 06:06:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:06:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 6, 40, 739427), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:06:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:06:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:06:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:06:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:06:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:42.941536+00:00 (in 1.792796 seconds) namespacegcworker stdout | 2023-01-17 06:06:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:41 UTC)" (scheduled at 2023-01-17 06:06:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:06:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:06:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 41, 149170), True, datetime.datetime(2023, 1, 17, 6, 6, 41, 149170), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:06:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:06:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:06:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:06:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:06:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:06.941185+00:00 (in 23.999164 seconds) namespacegcworker stdout | 2023-01-17 06:06:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:42 UTC)" (scheduled at 2023-01-17 06:06:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:06:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:06:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:07:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:55.646354+00:00 (in 9.999480 seconds) notificationworker stdout | 2023-01-17 06:06:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:55 UTC)" (scheduled at 2023-01-17 06:06:45.646354+00:00) notificationworker stdout | 2023-01-17 06:06:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 45, 647294), True, datetime.datetime(2023, 1, 17, 6, 6, 45, 647294), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:06:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:06:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:06:49.537472+00:00 (in 2.599991 seconds) securityworker stdout | 2023-01-17 06:06:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:16 UTC)" (scheduled at 2023-01-17 06:06:46.936988+00:00) securityworker stdout | 2023-01-17 06:06:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:16 UTC)" executed successfully builder stdout | 2023-01-17 06:06:47,847 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:47,859 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:47,859 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:47,870 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:06:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:16.936988+00:00 (in 27.399048 seconds) securityworker stdout | 2023-01-17 06:06:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:19 UTC)" (scheduled at 2023-01-17 06:06:49.537472+00:00) securityworker stdout | 2023-01-17 06:06:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:06:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:06:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:06:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:06:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:06:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:06:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:06:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:06:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:06:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:06:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:06:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:06:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:06:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:06:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:06:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:21.348605+00:00 (in 29.999548 seconds) gcworker stdout | 2023-01-17 06:06:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:21 UTC)" (scheduled at 2023-01-17 06:06:51.348605+00:00) gcworker stdout | 2023-01-17 06:06:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726011349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:06:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:06:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:06:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:06:52,060 [235] [DEBUG] [app] Starting request: urn:request:6cc1eb5b-0b3a-4f70-addf-a4834948ecbb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:52,060 [236] [DEBUG] [app] Starting request: urn:request:613c5e9f-54a2-4843-9ed4-44593df29960 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:06:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:06:52,065 [241] [DEBUG] [app] Starting request: urn:request:e8e8ba33-7ec8-4530-aef6-0fe3b168fc92 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:52,065 [240] [DEBUG] [app] Starting request: urn:request:0feaac33-bb34-425e-9589-5a5e12665b6c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:06:52,065 [241] [DEBUG] [app] Ending request: urn:request:e8e8ba33-7ec8-4530-aef6-0fe3b168fc92 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e8e8ba33-7ec8-4530-aef6-0fe3b168fc92', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:06:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:06:52,066 [240] [DEBUG] [app] Ending request: urn:request:0feaac33-bb34-425e-9589-5a5e12665b6c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0feaac33-bb34-425e-9589-5a5e12665b6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:06:52,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:06:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:06:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:06:52,070 [231] [DEBUG] [app] Starting request: urn:request:70c94aec-77f2-4f66-94b2-bae0f4ac252b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:52,070 [231] [DEBUG] [app] Ending request: urn:request:70c94aec-77f2-4f66-94b2-bae0f4ac252b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70c94aec-77f2-4f66-94b2-bae0f4ac252b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:52,071 [233] [DEBUG] [app] Starting request: urn:request:c9f4c52b-c4d1-45a0-ac47-7cfdd328f314 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:06:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:52,071 [233] [DEBUG] [app] Ending request: urn:request:c9f4c52b-c4d1-45a0-ac47-7cfdd328f314 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9f4c52b-c4d1-45a0-ac47-7cfdd328f314', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:06:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:06:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:06:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:06:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:06:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:06:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:06:52,087 [236] [DEBUG] [app] Ending request: urn:request:613c5e9f-54a2-4843-9ed4-44593df29960 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:613c5e9f-54a2-4843-9ed4-44593df29960', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:06:52,088 [235] [DEBUG] [app] Ending request: urn:request:6cc1eb5b-0b3a-4f70-addf-a4834948ecbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cc1eb5b-0b3a-4f70-addf-a4834948ecbb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:06:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:06:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:06:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) exportactionlogsworker stdout | 2023-01-17 06:06:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:06:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:28.963662+00:00 (in 33.999766 seconds) exportactionlogsworker stdout | 2023-01-17 06:06:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:54 UTC)" (scheduled at 2023-01-17 06:06:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 06:06:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 6, 54, 964326), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:06:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 54, 964326), True, datetime.datetime(2023, 1, 17, 6, 6, 54, 964326), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:06:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 6, 54, 964326), True, datetime.datetime(2023, 1, 17, 6, 6, 54, 964326), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 6, 54, 964326), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:06:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:06:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:11:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:06:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:06:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:01.636362+00:00 (in 5.989513 seconds) notificationworker stdout | 2023-01-17 06:06:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:05 UTC)" (scheduled at 2023-01-17 06:06:55.646354+00:00) notificationworker stdout | 2023-01-17 06:06:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:06:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 6, 55, 647250), True, datetime.datetime(2023, 1, 17, 6, 6, 55, 647250), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:06:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:06:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:06:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:05 UTC)" executed successfully builder stdout | 2023-01-17 06:06:57,880 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:06:57,893 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:06:57,893 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:06:57,903 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:00,638 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:07:00,721 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:07:01,556 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:07:01,559 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:07:01,561 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:07:01,563 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:07:01,566 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:07:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:05.646354+00:00 (in 4.009501 seconds) notificationworker stdout | 2023-01-17 06:07:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:01 UTC)" (scheduled at 2023-01-17 06:07:01.636362+00:00) notificationworker stdout | 2023-01-17 06:07:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:07:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:01 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:07:01,716 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:07:02,122 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:07:02,184 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:07:02,224 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:07:02,688 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:07:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:07:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:19.346120+00:00 (in 14.408841 seconds) repositorygcworker stdout | 2023-01-17 06:07:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:04 UTC)" (scheduled at 2023-01-17 06:07:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:07:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:07:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 4, 937735), True, datetime.datetime(2023, 1, 17, 6, 7, 4, 937735), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:07:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:07:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:07:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:07:05,545 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:07:05,547 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:07:05,549 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:07:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:15.646354+00:00 (in 9.999538 seconds) notificationworker stdout | 2023-01-17 06:07:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:15 UTC)" (scheduled at 2023-01-17 06:07:05.646354+00:00) notificationworker stdout | 2023-01-17 06:07:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 5, 647234), True, datetime.datetime(2023, 1, 17, 6, 7, 5, 647234), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:15 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:07:05,902 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:07:05,938 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:07:06,014 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:07:06,020 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:07:06,027 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:07:06,031 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:07:06,033 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:07:06,038 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:07:06,042 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:07:06,045 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:07:06,047 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} namespacegcworker stdout | 2023-01-17 06:07:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:07:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:41.148245+00:00 (in 34.206576 seconds) namespacegcworker stdout | 2023-01-17 06:07:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:12:06 UTC)" (scheduled at 2023-01-17 06:07:06.941185+00:00) namespacegcworker stdout | 2023-01-17 06:07:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 7, 6, 942047), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:07:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 7, 6, 942047), True, datetime.datetime(2023, 1, 17, 6, 7, 6, 942047), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:07:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 7, 6, 942047), True, datetime.datetime(2023, 1, 17, 6, 7, 6, 942047), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 7, 6, 942047), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:07:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:07:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:12:06 UTC)" executed successfully servicekey stdout | 2023-01-17 06:07:07,041 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 06:07:07,044 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 06:07:07,060 [231] [DEBUG] [app] Starting request: urn:request:e6ecfc6f-3dca-4cbb-9cc4-4007e40d8608 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:07:07,060 [236] [DEBUG] [app] Starting request: urn:request:c6ccd8df-f610-4364-8fe4-51af287b2541 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:07:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:07,064 [230] [DEBUG] [app] Starting request: urn:request:76820185-8136-4c04-8996-106d4a90f43e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:07,065 [242] [DEBUG] [app] Starting request: urn:request:0e222bdf-3b1b-43bc-97e6-5332844032db (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:07,065 [230] [DEBUG] [app] Ending request: urn:request:76820185-8136-4c04-8996-106d4a90f43e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76820185-8136-4c04-8996-106d4a90f43e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:07:07,065 [242] [DEBUG] [app] Ending request: urn:request:0e222bdf-3b1b-43bc-97e6-5332844032db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e222bdf-3b1b-43bc-97e6-5332844032db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:07:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:07:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:07,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:07,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:07,069 [233] [DEBUG] [app] Starting request: urn:request:24bf500d-b73b-4374-9b1a-9c3653ec5764 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:07,070 [233] [DEBUG] [app] Ending request: urn:request:24bf500d-b73b-4374-9b1a-9c3653ec5764 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:24bf500d-b73b-4374-9b1a-9c3653ec5764', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:07,070 [231] [DEBUG] [app] Starting request: urn:request:ec73d208-2db8-46a7-a0e5-9ede6bad49b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:07:07,070 [231] [DEBUG] [app] Ending request: urn:request:ec73d208-2db8-46a7-a0e5-9ede6bad49b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec73d208-2db8-46a7-a0e5-9ede6bad49b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:07,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:07,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:07,087 [231] [DEBUG] [app] Ending request: urn:request:e6ecfc6f-3dca-4cbb-9cc4-4007e40d8608 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6ecfc6f-3dca-4cbb-9cc4-4007e40d8608', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:07:07,087 [236] [DEBUG] [app] Ending request: urn:request:c6ccd8df-f610-4364-8fe4-51af287b2541 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c6ccd8df-f610-4364-8fe4-51af287b2541', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:07,087 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:07:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" repositorygcworker stdout | 2023-01-17 06:07:07,428 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} builder stdout | 2023-01-17 06:07:07,913 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:07,926 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:07,926 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:07,936 [60] [DEBUG] [data.database] Disconnecting from database. chunkcleanupworker stdout | 2023-01-17 06:07:07,940 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:07:08,050 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:07:08,150 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:07:08,389 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:07:08,912 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:07:09,416 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:07:09,529 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:07:09,782 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:07:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:07:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:40.738568+00:00 (in 29.999534 seconds) buildlogsarchiver stdout | 2023-01-17 06:07:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:40 UTC)" (scheduled at 2023-01-17 06:07:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:07:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 7, 10, 739445), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:07:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:07:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:07:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:07:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:25.646354+00:00 (in 9.999522 seconds) notificationworker stdout | 2023-01-17 06:07:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:25 UTC)" (scheduled at 2023-01-17 06:07:15.646354+00:00) notificationworker stdout | 2023-01-17 06:07:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 15, 647252), True, datetime.datetime(2023, 1, 17, 6, 7, 15, 647252), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:07:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:19.537472+00:00 (in 2.599984 seconds) securityworker stdout | 2023-01-17 06:07:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:46 UTC)" (scheduled at 2023-01-17 06:07:16.936988+00:00) securityworker stdout | 2023-01-17 06:07:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:46 UTC)" executed successfully builder stdout | 2023-01-17 06:07:17,939 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:17,952 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:17,952 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:17,962 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:07:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:07:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:04.936835+00:00 (in 45.590203 seconds) repositorygcworker stdout | 2023-01-17 06:07:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:19 UTC)" (scheduled at 2023-01-17 06:07:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:07:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:07:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:07:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:46.936988+00:00 (in 27.399029 seconds) securityworker stdout | 2023-01-17 06:07:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:49 UTC)" (scheduled at 2023-01-17 06:07:19.537472+00:00) securityworker stdout | 2023-01-17 06:07:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:07:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:07:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:19,565 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:19,565 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:07:19,565 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:07:19,565 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:07:19,565 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:19,566 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:19,566 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:19,566 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:07:19,566 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:07:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:07:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:51.348605+00:00 (in 29.999565 seconds) gcworker stdout | 2023-01-17 06:07:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:51 UTC)" (scheduled at 2023-01-17 06:07:21.348605+00:00) gcworker stdout | 2023-01-17 06:07:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726041349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:07:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:07:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:07:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:07:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:07:22,060 [236] [DEBUG] [app] Starting request: urn:request:dca59b01-0508-4c06-8c5a-3ae6d9950ba8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:07:22,060 [235] [DEBUG] [app] Starting request: urn:request:e5744825-f60c-4d20-ae3f-f1e6fa055ab7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:07:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:22,065 [242] [DEBUG] [app] Starting request: urn:request:c31937a5-20cb-4e79-b060-cacf391cbfb4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:22,065 [242] [DEBUG] [app] Ending request: urn:request:c31937a5-20cb-4e79-b060-cacf391cbfb4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c31937a5-20cb-4e79-b060-cacf391cbfb4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:07:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:07:22,065 [230] [DEBUG] [app] Starting request: urn:request:4081869f-08c3-4f40-9b75-341a1d70c921 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:07:22,066 [230] [DEBUG] [app] Ending request: urn:request:4081869f-08c3-4f40-9b75-341a1d70c921 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4081869f-08c3-4f40-9b75-341a1d70c921', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:07:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:22,071 [231] [DEBUG] [app] Starting request: urn:request:05d1c79e-1abf-4acb-a3af-85b9b8cce6c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:22,071 [236] [DEBUG] [app] Starting request: urn:request:f4870164-9e1f-4078-a511-ad55a0001ed6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:22,071 [231] [DEBUG] [app] Ending request: urn:request:05d1c79e-1abf-4acb-a3af-85b9b8cce6c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05d1c79e-1abf-4acb-a3af-85b9b8cce6c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:22,071 [236] [DEBUG] [app] Ending request: urn:request:f4870164-9e1f-4078-a511-ad55a0001ed6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4870164-9e1f-4078-a511-ad55a0001ed6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:22,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:22,088 [236] [DEBUG] [app] Ending request: urn:request:dca59b01-0508-4c06-8c5a-3ae6d9950ba8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dca59b01-0508-4c06-8c5a-3ae6d9950ba8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:07:22,088 [235] [DEBUG] [app] Ending request: urn:request:e5744825-f60c-4d20-ae3f-f1e6fa055ab7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5744825-f60c-4d20-ae3f-f1e6fa055ab7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:07:22,089 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) notificationworker stdout | 2023-01-17 06:07:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:35.646354+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 06:07:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:35 UTC)" (scheduled at 2023-01-17 06:07:25.646354+00:00) notificationworker stdout | 2023-01-17 06:07:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 25, 647284), True, datetime.datetime(2023, 1, 17, 6, 7, 25, 647284), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:35 UTC)" executed successfully builder stdout | 2023-01-17 06:07:27,972 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:27,985 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:27,985 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:27,996 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:07:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:07:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:35.853396+00:00 (in 6.889204 seconds) exportactionlogsworker stdout | 2023-01-17 06:07:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:28 UTC)" (scheduled at 2023-01-17 06:07:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:07:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:07:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:07:30,670 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:07:30,756 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:07:31,564 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:07:31,567 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:07:31,569 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:07:31,571 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:07:31,573 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:07:31,740 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:07:32,128 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:07:32,219 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:07:32,237 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:07:32,723 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:07:35,553 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:07:35,555 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:07:35,558 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:07:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:45.646354+00:00 (in 9.999517 seconds) notificationworker stdout | 2023-01-17 06:07:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:45 UTC)" (scheduled at 2023-01-17 06:07:35.646354+00:00) notificationworker stdout | 2023-01-17 06:07:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 35, 647252), True, datetime.datetime(2023, 1, 17, 6, 7, 35, 647252), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:07:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:07:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:28.963662+00:00 (in 53.109758 seconds) exportactionlogsworker stdout | 2023-01-17 06:07:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:35 UTC)" (scheduled at 2023-01-17 06:07:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:07:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:07:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 35, 854327), True, datetime.datetime(2023, 1, 17, 6, 7, 35, 854327), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:07:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:07:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:07:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:35 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:07:35,912 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:07:35,973 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:07:36,021 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:07:36,029 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:07:36,036 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:07:36,039 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:07:36,041 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:07:36,044 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:07:36,050 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:07:36,052 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:07:36,055 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:07:37,059 [236] [DEBUG] [app] Starting request: urn:request:d004156c-9038-45a6-b9b5-692ee3b3ba93 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:07:37,060 [231] [DEBUG] [app] Starting request: urn:request:5d9315e4-f243-4823-b7a0-1e9762fc2444 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} storagereplication stdout | 2023-01-17 06:07:37,059 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 06:07:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:37,065 [242] [DEBUG] [app] Starting request: urn:request:c3139b4e-d512-40ad-874a-67ad0943b72a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:37,065 [243] [DEBUG] [app] Starting request: urn:request:60358d2f-5e12-451f-8872-3ca26d2ea88d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:37,065 [242] [DEBUG] [app] Ending request: urn:request:c3139b4e-d512-40ad-874a-67ad0943b72a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3139b4e-d512-40ad-874a-67ad0943b72a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:37,065 [243] [DEBUG] [app] Ending request: urn:request:60358d2f-5e12-451f-8872-3ca26d2ea88d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60358d2f-5e12-451f-8872-3ca26d2ea88d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:07:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:07:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:37,070 [231] [DEBUG] [app] Starting request: urn:request:130537db-6bb8-43ff-aec7-f513bb7a0298 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:37,070 [235] [DEBUG] [app] Starting request: urn:request:bd11ef8a-8e59-4a95-aea6-b7c46d8760ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:37,071 [231] [DEBUG] [app] Ending request: urn:request:130537db-6bb8-43ff-aec7-f513bb7a0298 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:130537db-6bb8-43ff-aec7-f513bb7a0298', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:37,071 [235] [DEBUG] [app] Ending request: urn:request:bd11ef8a-8e59-4a95-aea6-b7c46d8760ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bd11ef8a-8e59-4a95-aea6-b7c46d8760ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:07:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql servicekey stdout | 2023-01-17 06:07:37,076 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:07:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:37,088 [231] [DEBUG] [app] Ending request: urn:request:5d9315e4-f243-4823-b7a0-1e9762fc2444 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d9315e4-f243-4823-b7a0-1e9762fc2444', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:37,088 [236] [DEBUG] [app] Ending request: urn:request:d004156c-9038-45a6-b9b5-692ee3b3ba93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d004156c-9038-45a6-b9b5-692ee3b3ba93', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:07:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 06:07:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:07:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:37.537407+00:00 (in 0.280333 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:07:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:37 UTC)" (scheduled at 2023-01-17 06:07:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:07:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:07:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 37, 257488), True, datetime.datetime(2023, 1, 17, 6, 7, 37, 257488), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:07:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:07:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:07:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:07:37,454 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} securityscanningnotificationworker stdout | 2023-01-17 06:07:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:07:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:37.256652+00:00 (in 59.718831 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:07:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:37 UTC)" (scheduled at 2023-01-17 06:07:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:07:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:07:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 06:07:37,973 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} builder stdout | 2023-01-17 06:07:38,006 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:38,018 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:38,018 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:38,028 [60] [DEBUG] [data.database] Disconnecting from database. manifestbackfillworker stdout | 2023-01-17 06:07:38,084 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:07:38,184 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:07:38,409 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:07:38,932 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:07:39,421 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:07:39,550 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:07:39,813 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:07:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:07:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:10.738568+00:00 (in 29.999511 seconds) buildlogsarchiver stdout | 2023-01-17 06:07:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:10 UTC)" (scheduled at 2023-01-17 06:07:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:07:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 7, 40, 739474), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:07:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:07:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:07:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:07:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:07:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:42.941536+00:00 (in 1.792811 seconds) namespacegcworker stdout | 2023-01-17 06:07:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:41 UTC)" (scheduled at 2023-01-17 06:07:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:07:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:07:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 41, 149118), True, datetime.datetime(2023, 1, 17, 6, 7, 41, 149118), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:07:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:07:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:07:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:07:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:07:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:41.148245+00:00 (in 58.206233 seconds) namespacegcworker stdout | 2023-01-17 06:07:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:42 UTC)" (scheduled at 2023-01-17 06:07:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:07:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:07:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:08:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:07:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:55.646354+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 06:07:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:55 UTC)" (scheduled at 2023-01-17 06:07:45.646354+00:00) notificationworker stdout | 2023-01-17 06:07:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 45, 647170), True, datetime.datetime(2023, 1, 17, 6, 7, 45, 647170), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:07:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:07:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:07:49.537472+00:00 (in 2.600026 seconds) securityworker stdout | 2023-01-17 06:07:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:16 UTC)" (scheduled at 2023-01-17 06:07:46.936988+00:00) securityworker stdout | 2023-01-17 06:07:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:16 UTC)" executed successfully builder stdout | 2023-01-17 06:07:48,038 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:48,051 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:48,051 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:48,062 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:07:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:16.936988+00:00 (in 27.399036 seconds) securityworker stdout | 2023-01-17 06:07:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:19 UTC)" (scheduled at 2023-01-17 06:07:49.537472+00:00) securityworker stdout | 2023-01-17 06:07:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:07:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:07:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:07:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:07:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:07:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:07:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:07:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:07:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:07:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:07:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:07:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:07:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:07:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:07:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:07:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:21.348605+00:00 (in 29.999537 seconds) gcworker stdout | 2023-01-17 06:07:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:21 UTC)" (scheduled at 2023-01-17 06:07:51.348605+00:00) gcworker stdout | 2023-01-17 06:07:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726071349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:07:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:07:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:07:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:07:52,060 [236] [DEBUG] [app] Starting request: urn:request:f87d9923-d530-49e3-a473-07b3dd0abde8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:07:52,060 [235] [DEBUG] [app] Starting request: urn:request:fbc92395-b840-4270-b41a-85da3e2885e3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:07:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:07:52,065 [243] [DEBUG] [app] Starting request: urn:request:2aca0208-4ca1-4e2d-ae16-a36b06780a2c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:52,065 [230] [DEBUG] [app] Starting request: urn:request:43840c58-6dc8-4940-8990-c288888911a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:07:52,065 [243] [DEBUG] [app] Ending request: urn:request:2aca0208-4ca1-4e2d-ae16-a36b06780a2c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2aca0208-4ca1-4e2d-ae16-a36b06780a2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:07:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:07:52,066 [230] [DEBUG] [app] Ending request: urn:request:43840c58-6dc8-4940-8990-c288888911a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:43840c58-6dc8-4940-8990-c288888911a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:07:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:07:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:07:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:52,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:07:52,070 [233] [DEBUG] [app] Starting request: urn:request:1522bbf6-687e-4ad3-b326-ed0c15ea45af (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:52,071 [231] [DEBUG] [app] Starting request: urn:request:f856efb9-81d8-44c7-b78e-b12ed269162d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:07:52,071 [233] [DEBUG] [app] Ending request: urn:request:1522bbf6-687e-4ad3-b326-ed0c15ea45af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1522bbf6-687e-4ad3-b326-ed0c15ea45af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:52,071 [231] [DEBUG] [app] Ending request: urn:request:f856efb9-81d8-44c7-b78e-b12ed269162d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f856efb9-81d8-44c7-b78e-b12ed269162d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:07:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:07:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:07:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:07:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:07:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:52,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:07:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:07:52,088 [235] [DEBUG] [app] Ending request: urn:request:fbc92395-b840-4270-b41a-85da3e2885e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fbc92395-b840-4270-b41a-85da3e2885e3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:52,088 [236] [DEBUG] [app] Ending request: urn:request:f87d9923-d530-49e3-a473-07b3dd0abde8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f87d9923-d530-49e3-a473-07b3dd0abde8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:07:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:07:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:07:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:07:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:07:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:07:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:01.636362+00:00 (in 5.989569 seconds) notificationworker stdout | 2023-01-17 06:07:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:05 UTC)" (scheduled at 2023-01-17 06:07:55.646354+00:00) notificationworker stdout | 2023-01-17 06:07:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:07:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 7, 55, 647173), True, datetime.datetime(2023, 1, 17, 6, 7, 55, 647173), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:07:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:07:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:07:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:05 UTC)" executed successfully builder stdout | 2023-01-17 06:07:58,062 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:07:58,075 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:07:58,075 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:07:58,085 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:00,687 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:08:00,761 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:08:01,571 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:08:01,574 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:08:01,577 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:08:01,580 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:08:01,582 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:08:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:05.646354+00:00 (in 4.009534 seconds) notificationworker stdout | 2023-01-17 06:08:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:01 UTC)" (scheduled at 2023-01-17 06:08:01.636362+00:00) notificationworker stdout | 2023-01-17 06:08:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:08:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:01 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:08:01,775 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:08:02,147 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:08:02,238 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:08:02,257 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:08:02,729 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:08:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:08:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:19.346120+00:00 (in 14.408860 seconds) repositorygcworker stdout | 2023-01-17 06:08:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:04 UTC)" (scheduled at 2023-01-17 06:08:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:08:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:08:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 4, 937717), True, datetime.datetime(2023, 1, 17, 6, 8, 4, 937717), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:08:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:08:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:08:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:08:05,560 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:08:05,563 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:08:05,565 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:08:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:15.646354+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 06:08:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:15 UTC)" (scheduled at 2023-01-17 06:08:05.646354+00:00) notificationworker stdout | 2023-01-17 06:08:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 5, 647207), True, datetime.datetime(2023, 1, 17, 6, 8, 5, 647207), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:15 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:08:05,947 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:08:06,008 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:08:06,027 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:08:06,037 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:08:06,044 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:08:06,048 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:08:06,050 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:08:06,052 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:08:06,058 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:08:06,061 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:08:06,063 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:08:07,060 [231] [DEBUG] [app] Starting request: urn:request:2d65e10b-61b3-4ec4-85e1-519daa63c7b0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:07,060 [236] [DEBUG] [app] Starting request: urn:request:e2f6b2d0-91d3-4097-a880-76df8ebc4f40 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:07,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:07,065 [243] [DEBUG] [app] Starting request: urn:request:857caa4e-9428-4815-a893-239461969fd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:07,066 [230] [DEBUG] [app] Starting request: urn:request:411885e3-76c1-4ebf-8974-53e0c504126a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:07,066 [243] [DEBUG] [app] Ending request: urn:request:857caa4e-9428-4815-a893-239461969fd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:857caa4e-9428-4815-a893-239461969fd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:08:07,066 [230] [DEBUG] [app] Ending request: urn:request:411885e3-76c1-4ebf-8974-53e0c504126a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:411885e3-76c1-4ebf-8974-53e0c504126a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:08:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:08:07,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:07,071 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:07,071 [233] [DEBUG] [app] Starting request: urn:request:66e0c8b9-392b-4d8c-b9fd-6b0fcc17c9ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:07,071 [231] [DEBUG] [app] Starting request: urn:request:94f83a1e-a2f0-49c4-85ce-35bff9ec22e9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:07,071 [233] [DEBUG] [app] Ending request: urn:request:66e0c8b9-392b-4d8c-b9fd-6b0fcc17c9ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66e0c8b9-392b-4d8c-b9fd-6b0fcc17c9ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:07,071 [231] [DEBUG] [app] Ending request: urn:request:94f83a1e-a2f0-49c4-85ce-35bff9ec22e9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:94f83a1e-a2f0-49c4-85ce-35bff9ec22e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) servicekey stdout | 2023-01-17 06:08:07,082 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:08:07,088 [231] [DEBUG] [app] Ending request: urn:request:2d65e10b-61b3-4ec4-85e1-519daa63c7b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d65e10b-61b3-4ec4-85e1-519daa63c7b0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:08:07,089 [236] [DEBUG] [app] Ending request: urn:request:e2f6b2d0-91d3-4097-a880-76df8ebc4f40 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e2f6b2d0-91d3-4097-a880-76df8ebc4f40', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 06:08:07,089 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:07,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) storagereplication stdout | 2023-01-17 06:08:07,093 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 06:08:07,475 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:08:07,984 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} builder stdout | 2023-01-17 06:08:08,096 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue manifestbackfillworker stdout | 2023-01-17 06:08:08,106 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:08:08,108 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:08,108 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:08,119 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:08:08,202 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:08:08,421 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:08:08,960 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:08:09,457 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:08:09,586 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:08:09,828 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:08:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:08:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:40.738568+00:00 (in 29.999527 seconds) buildlogsarchiver stdout | 2023-01-17 06:08:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:40 UTC)" (scheduled at 2023-01-17 06:08:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:08:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 8, 10, 739416), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:08:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:08:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:08:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:08:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:25.646354+00:00 (in 9.999521 seconds) notificationworker stdout | 2023-01-17 06:08:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:25 UTC)" (scheduled at 2023-01-17 06:08:15.646354+00:00) notificationworker stdout | 2023-01-17 06:08:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 15, 647255), True, datetime.datetime(2023, 1, 17, 6, 8, 15, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:08:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:19.537472+00:00 (in 2.599985 seconds) securityworker stdout | 2023-01-17 06:08:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:46 UTC)" (scheduled at 2023-01-17 06:08:16.936988+00:00) securityworker stdout | 2023-01-17 06:08:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:46 UTC)" executed successfully builder stdout | 2023-01-17 06:08:18,124 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:18,137 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:18,137 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:18,148 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:08:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:08:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:04.936835+00:00 (in 45.590265 seconds) repositorygcworker stdout | 2023-01-17 06:08:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:19 UTC)" (scheduled at 2023-01-17 06:08:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:08:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:08:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:08:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:46.936988+00:00 (in 27.399023 seconds) securityworker stdout | 2023-01-17 06:08:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:49 UTC)" (scheduled at 2023-01-17 06:08:19.537472+00:00) securityworker stdout | 2023-01-17 06:08:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:08:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:08:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:08:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:08:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:08:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:08:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:08:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:08:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:51.348605+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 06:08:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:51 UTC)" (scheduled at 2023-01-17 06:08:21.348605+00:00) gcworker stdout | 2023-01-17 06:08:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726101349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:08:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:08:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:08:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:08:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:08:22,060 [233] [DEBUG] [app] Starting request: urn:request:18c85529-28ef-498f-9a6b-4f79ca3e5582 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:22,060 [236] [DEBUG] [app] Starting request: urn:request:cb22a13a-11f2-44e9-ab69-9f2dfc180936 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:22,065 [239] [DEBUG] [app] Starting request: urn:request:59896110-5f26-4eb6-a7c3-ab416f12606e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:22,065 [230] [DEBUG] [app] Starting request: urn:request:86308769-4603-4966-8af8-c18a6440776a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:22,066 [239] [DEBUG] [app] Ending request: urn:request:59896110-5f26-4eb6-a7c3-ab416f12606e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59896110-5f26-4eb6-a7c3-ab416f12606e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:22,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:08:22,066 [230] [DEBUG] [app] Ending request: urn:request:86308769-4603-4966-8af8-c18a6440776a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86308769-4603-4966-8af8-c18a6440776a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:08:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:08:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:22,070 [231] [DEBUG] [app] Starting request: urn:request:cc7c3738-df2d-45c6-8c5f-6e1df2198b91 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:22,071 [231] [DEBUG] [app] Ending request: urn:request:cc7c3738-df2d-45c6-8c5f-6e1df2198b91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cc7c3738-df2d-45c6-8c5f-6e1df2198b91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:22,071 [235] [DEBUG] [app] Starting request: urn:request:bb10887b-1ca5-4079-8eca-87474adfcdfa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:22,071 [235] [DEBUG] [app] Ending request: urn:request:bb10887b-1ca5-4079-8eca-87474adfcdfa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb10887b-1ca5-4079-8eca-87474adfcdfa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:08:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:22,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:22,087 [233] [DEBUG] [app] Ending request: urn:request:18c85529-28ef-498f-9a6b-4f79ca3e5582 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:18c85529-28ef-498f-9a6b-4f79ca3e5582', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:22,087 [236] [DEBUG] [app] Ending request: urn:request:cb22a13a-11f2-44e9-ab69-9f2dfc180936 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb22a13a-11f2-44e9-ab69-9f2dfc180936', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:08:22,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:08:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:08:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:35.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 06:08:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:35 UTC)" (scheduled at 2023-01-17 06:08:25.646354+00:00) notificationworker stdout | 2023-01-17 06:08:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 25, 647186), True, datetime.datetime(2023, 1, 17, 6, 8, 25, 647186), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:35 UTC)" executed successfully builder stdout | 2023-01-17 06:08:28,158 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:28,171 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:28,171 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:28,181 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:08:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:08:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:35.853396+00:00 (in 6.889271 seconds) exportactionlogsworker stdout | 2023-01-17 06:08:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:28 UTC)" (scheduled at 2023-01-17 06:08:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:08:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:08:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:08:30,715 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:08:30,790 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:08:31,578 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:08:31,581 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:08:31,584 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:08:31,587 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:08:31,589 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:08:31,811 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:08:32,183 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:08:32,273 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:08:32,286 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:08:32,755 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:08:35,568 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:08:35,570 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:08:35,572 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:08:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:36.635997+00:00 (in 0.989213 seconds) notificationworker stdout | 2023-01-17 06:08:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:45 UTC)" (scheduled at 2023-01-17 06:08:35.646354+00:00) notificationworker stdout | 2023-01-17 06:08:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 35, 647199), True, datetime.datetime(2023, 1, 17, 6, 8, 35, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:08:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:08:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:28.963662+00:00 (in 53.109746 seconds) exportactionlogsworker stdout | 2023-01-17 06:08:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:35 UTC)" (scheduled at 2023-01-17 06:08:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:08:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:08:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 35, 854297), True, datetime.datetime(2023, 1, 17, 6, 8, 35, 854297), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:08:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:08:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:08:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:35 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:08:35,953 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-registry stdout | 2023-01-17 06:08:36,038 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} expiredappspecifictokenworker stdout | 2023-01-17 06:08:36,043 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:08:36,046 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:08:36,052 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:08:36,054 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:08:36,057 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:08:36,059 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:08:36,065 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:08:36,068 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:08:36,070 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} notificationworker stdout | 2023-01-17 06:08:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:45.646354+00:00 (in 9.009909 seconds) notificationworker stdout | 2023-01-17 06:08:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:13:36 UTC)" (scheduled at 2023-01-17 06:08:36.635997+00:00) notificationworker stdout | 2023-01-17 06:08:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 8, 36, 636823), 'notification/%']) notificationworker stdout | 2023-01-17 06:08:36,648 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 8, 36, 636823), True, datetime.datetime(2023, 1, 17, 6, 8, 36, 636823), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:08:36,652 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 8, 36, 636823), True, datetime.datetime(2023, 1, 17, 6, 8, 36, 636823), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 8, 36, 636823), 'notification/%']) notificationworker stdout | 2023-01-17 06:08:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:13:36 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:08:37,060 [231] [DEBUG] [app] Starting request: urn:request:a6d53fc5-12b7-40f3-9cdc-f0ffe8b46d45 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:37,060 [235] [DEBUG] [app] Starting request: urn:request:dcd52280-892f-4250-897f-b145463a22da (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:37,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:37,065 [243] [DEBUG] [app] Starting request: urn:request:e97f0ffe-2533-4b7b-8a97-014c66bfe078 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:37,066 [243] [DEBUG] [app] Ending request: urn:request:e97f0ffe-2533-4b7b-8a97-014c66bfe078 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e97f0ffe-2533-4b7b-8a97-014c66bfe078', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:37,066 [230] [DEBUG] [app] Starting request: urn:request:6d623ae4-d5a2-452e-9ed8-8e32dc021571 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:08:37,066 [230] [DEBUG] [app] Ending request: urn:request:6d623ae4-d5a2-452e-9ed8-8e32dc021571 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d623ae4-d5a2-452e-9ed8-8e32dc021571', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:08:37,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:08:37,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,069 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:37,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:37,071 [231] [DEBUG] [app] Starting request: urn:request:ae64a5a7-aeb5-4647-97e4-451c47c03d7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:37,071 [231] [DEBUG] [app] Ending request: urn:request:ae64a5a7-aeb5-4647-97e4-451c47c03d7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae64a5a7-aeb5-4647-97e4-451c47c03d7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:37,072 [233] [DEBUG] [app] Starting request: urn:request:0d1ea996-74e7-4af9-8893-cb73dedf48ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:37,072 [233] [DEBUG] [app] Ending request: urn:request:0d1ea996-74e7-4af9-8893-cb73dedf48ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d1ea996-74e7-4af9-8893-cb73dedf48ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:08:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:37,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) servicekey stdout | 2023-01-17 06:08:37,101 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 06:08:37,121 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 06:08:37,125 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:37,134 [231] [DEBUG] [app] Ending request: urn:request:a6d53fc5-12b7-40f3-9cdc-f0ffe8b46d45 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6d53fc5-12b7-40f3-9cdc-f0ffe8b46d45', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:37,134 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:37,134 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.075 47 0.075) gunicorn-web stdout | 2023-01-17 06:08:37,159 [235] [DEBUG] [app] Ending request: urn:request:dcd52280-892f-4250-897f-b145463a22da (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcd52280-892f-4250-897f-b145463a22da', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:37,159 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:37,159 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.100 47 0.100) securityscanningnotificationworker stdout | 2023-01-17 06:08:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:08:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:37.537407+00:00 (in 0.280301 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:08:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:37 UTC)" (scheduled at 2023-01-17 06:08:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:08:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:08:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 37, 257612), True, datetime.datetime(2023, 1, 17, 6, 8, 37, 257612), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:08:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:08:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:08:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:08:37,503 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} securityscanningnotificationworker stdout | 2023-01-17 06:08:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:08:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:37.256652+00:00 (in 59.718774 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:08:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:37 UTC)" (scheduled at 2023-01-17 06:08:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:08:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:08:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:37 UTC)" executed successfully chunkcleanupworker stdout | 2023-01-17 06:08:38,020 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:08:38,133 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} builder stdout | 2023-01-17 06:08:38,192 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:38,204 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:38,204 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:38,215 [60] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:08:38,218 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:08:38,430 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:08:38,997 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:08:39,481 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:08:39,593 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:08:39,844 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:08:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:08:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:10.738568+00:00 (in 29.999543 seconds) buildlogsarchiver stdout | 2023-01-17 06:08:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:10 UTC)" (scheduled at 2023-01-17 06:08:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:08:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 8, 40, 739449), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:08:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:08:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:08:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:08:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:08:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:42.941536+00:00 (in 1.792835 seconds) namespacegcworker stdout | 2023-01-17 06:08:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:41 UTC)" (scheduled at 2023-01-17 06:08:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:08:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:08:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 41, 149118), True, datetime.datetime(2023, 1, 17, 6, 8, 41, 149118), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:08:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:08:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:08:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:08:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:08:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:41.148245+00:00 (in 58.206257 seconds) namespacegcworker stdout | 2023-01-17 06:08:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:42 UTC)" (scheduled at 2023-01-17 06:08:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:08:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:08:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:09:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:08:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:55.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 06:08:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:55 UTC)" (scheduled at 2023-01-17 06:08:45.646354+00:00) notificationworker stdout | 2023-01-17 06:08:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 45, 647264), True, datetime.datetime(2023, 1, 17, 6, 8, 45, 647264), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:08:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:08:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:08:49.537472+00:00 (in 2.599983 seconds) securityworker stdout | 2023-01-17 06:08:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:16 UTC)" (scheduled at 2023-01-17 06:08:46.936988+00:00) securityworker stdout | 2023-01-17 06:08:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:16 UTC)" executed successfully builder stdout | 2023-01-17 06:08:48,222 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:48,235 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:48,235 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:48,245 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:08:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:16.936988+00:00 (in 27.398989 seconds) securityworker stdout | 2023-01-17 06:08:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:19 UTC)" (scheduled at 2023-01-17 06:08:49.537472+00:00) securityworker stdout | 2023-01-17 06:08:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:08:49,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:49,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:08:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:08:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:08:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:08:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:08:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:08:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:08:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:08:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:08:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:08:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:08:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:08:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:08:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:08:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:21.348605+00:00 (in 29.999554 seconds) gcworker stdout | 2023-01-17 06:08:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:21 UTC)" (scheduled at 2023-01-17 06:08:51.348605+00:00) gcworker stdout | 2023-01-17 06:08:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726131349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:08:51,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:08:51,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:08:51,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:08:52,060 [233] [DEBUG] [app] Starting request: urn:request:6334102a-3adb-417d-9bb1-25f9856ce6c2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:52,060 [235] [DEBUG] [app] Starting request: urn:request:6f9a681a-27ae-4d5a-af9f-24d0d16702d8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:08:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:08:52,065 [243] [DEBUG] [app] Starting request: urn:request:1e6b28ee-e17a-4e57-b5cf-18a01202c1fd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:08:52,065 [243] [DEBUG] [app] Ending request: urn:request:1e6b28ee-e17a-4e57-b5cf-18a01202c1fd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e6b28ee-e17a-4e57-b5cf-18a01202c1fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:08:52,065 [242] [DEBUG] [app] Starting request: urn:request:469018ac-66a0-49c5-89c5-ee85e8dee40a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:08:52,066 [242] [DEBUG] [app] Ending request: urn:request:469018ac-66a0-49c5-89c5-ee85e8dee40a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:469018ac-66a0-49c5-89c5-ee85e8dee40a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:08:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:08:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:08:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:52,069 [235] [DEBUG] [app] Starting request: urn:request:c702d617-8de0-40ae-a63b-1be4706bffcf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:52,070 [235] [DEBUG] [app] Ending request: urn:request:c702d617-8de0-40ae-a63b-1be4706bffcf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c702d617-8de0-40ae-a63b-1be4706bffcf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:08:52,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:52,071 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:08:52,072 [236] [DEBUG] [app] Starting request: urn:request:d333177e-40cf-4a35-883a-daf73cd24717 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:08:52,072 [236] [DEBUG] [app] Ending request: urn:request:d333177e-40cf-4a35-883a-daf73cd24717 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d333177e-40cf-4a35-883a-daf73cd24717', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:08:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:08:52,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:08:52,073 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:08:52,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:52,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:08:52,086 [233] [DEBUG] [app] Ending request: urn:request:6334102a-3adb-417d-9bb1-25f9856ce6c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6334102a-3adb-417d-9bb1-25f9856ce6c2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:52,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:52,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:08:52,088 [235] [DEBUG] [app] Ending request: urn:request:6f9a681a-27ae-4d5a-af9f-24d0d16702d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f9a681a-27ae-4d5a-af9f-24d0d16702d8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:08:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:08:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:08:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:08:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:08:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:01.636362+00:00 (in 5.989559 seconds) notificationworker stdout | 2023-01-17 06:08:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:05 UTC)" (scheduled at 2023-01-17 06:08:55.646354+00:00) notificationworker stdout | 2023-01-17 06:08:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:08:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 8, 55, 647217), True, datetime.datetime(2023, 1, 17, 6, 8, 55, 647217), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:08:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:08:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:08:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:05 UTC)" executed successfully builder stdout | 2023-01-17 06:08:58,249 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:08:58,262 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:08:58,262 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:08:58,273 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:00,748 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:09:00,805 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:09:01,586 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:09:01,589 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:09:01,591 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:09:01,594 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:09:01,596 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:09:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:05.646354+00:00 (in 4.009532 seconds) notificationworker stdout | 2023-01-17 06:09:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:01 UTC)" (scheduled at 2023-01-17 06:09:01.636362+00:00) notificationworker stdout | 2023-01-17 06:09:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:09:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:01 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:09:01,847 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:09:02,189 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:09:02,309 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:09:02,311 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:09:02,774 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:09:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:09:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:19.346120+00:00 (in 14.408833 seconds) repositorygcworker stdout | 2023-01-17 06:09:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:04 UTC)" (scheduled at 2023-01-17 06:09:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:09:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:09:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 4, 937708), True, datetime.datetime(2023, 1, 17, 6, 9, 4, 937708), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:09:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:09:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:09:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:09:05,574 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:09:05,576 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:09:05,578 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:09:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:15.646354+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 06:09:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:15 UTC)" (scheduled at 2023-01-17 06:09:05.646354+00:00) notificationworker stdout | 2023-01-17 06:09:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 5, 647195), True, datetime.datetime(2023, 1, 17, 6, 9, 5, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:05,665 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:05,665 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:05,666 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:15 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:09:05,987 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-registry stdout | 2023-01-17 06:09:06,048 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} expiredappspecifictokenworker stdout | 2023-01-17 06:09:06,051 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:09:06,053 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:09:06,058 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:09:06,061 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:09:06,064 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:09:06,066 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:09:06,072 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:09:06,074 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:09:06,076 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:09:07,060 [231] [DEBUG] [app] Starting request: urn:request:562c99aa-64fa-421d-adec-e0936502f108 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:07,060 [235] [DEBUG] [app] Starting request: urn:request:a80339bd-f328-4ab5-a9a7-1792cb5eb23d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:07,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:07,065 [243] [DEBUG] [app] Starting request: urn:request:4e503611-75b9-4bc3-a886-1941212d177d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:07,065 [243] [DEBUG] [app] Ending request: urn:request:4e503611-75b9-4bc3-a886-1941212d177d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e503611-75b9-4bc3-a886-1941212d177d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:09:07,065 [242] [DEBUG] [app] Starting request: urn:request:6a2f90f0-1fe8-4466-ac98-7d13d56e5a1f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:09:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:09:07,066 [242] [DEBUG] [app] Ending request: urn:request:6a2f90f0-1fe8-4466-ac98-7d13d56e5a1f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6a2f90f0-1fe8-4466-ac98-7d13d56e5a1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:09:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:07,070 [236] [DEBUG] [app] Starting request: urn:request:1e99883d-d282-49c2-9f7a-0c01b65895cd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:07,070 [236] [DEBUG] [app] Ending request: urn:request:1e99883d-d282-49c2-9f7a-0c01b65895cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e99883d-d282-49c2-9f7a-0c01b65895cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:07,070 [233] [DEBUG] [app] Starting request: urn:request:7c298e8b-0ce8-4081-bfdc-bccacb5e7b75 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:07,071 [233] [DEBUG] [app] Ending request: urn:request:7c298e8b-0ce8-4081-bfdc-bccacb5e7b75 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c298e8b-0ce8-4081-bfdc-bccacb5e7b75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:09:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:07,087 [231] [DEBUG] [app] Ending request: urn:request:562c99aa-64fa-421d-adec-e0936502f108 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:562c99aa-64fa-421d-adec-e0936502f108', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:07,087 [235] [DEBUG] [app] Ending request: urn:request:a80339bd-f328-4ab5-a9a7-1792cb5eb23d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a80339bd-f328-4ab5-a9a7-1792cb5eb23d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:09:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.027 47 0.028) servicekey stdout | 2023-01-17 06:09:07,136 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 06:09:07,148 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 06:09:07,538 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:09:08,054 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:09:08,156 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:09:08,237 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} builder stdout | 2023-01-17 06:09:08,283 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:08,296 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:08,296 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:08,307 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:09:08,466 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:09:09,022 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:09:09,490 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:09:09,629 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:09:09,880 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:09:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:09:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:40.738568+00:00 (in 29.999517 seconds) buildlogsarchiver stdout | 2023-01-17 06:09:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:40 UTC)" (scheduled at 2023-01-17 06:09:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:09:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 9, 10, 739456), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:09:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:09:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:09:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:09:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:25.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 06:09:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:25 UTC)" (scheduled at 2023-01-17 06:09:15.646354+00:00) notificationworker stdout | 2023-01-17 06:09:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 15, 647235), True, datetime.datetime(2023, 1, 17, 6, 9, 15, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:09:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:19.537472+00:00 (in 2.599972 seconds) securityworker stdout | 2023-01-17 06:09:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:46 UTC)" (scheduled at 2023-01-17 06:09:16.936988+00:00) securityworker stdout | 2023-01-17 06:09:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:46 UTC)" executed successfully builder stdout | 2023-01-17 06:09:18,317 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:18,330 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:18,330 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:18,341 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:09:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:09:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:04.936835+00:00 (in 45.590277 seconds) repositorygcworker stdout | 2023-01-17 06:09:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:19 UTC)" (scheduled at 2023-01-17 06:09:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:09:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:09:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:09:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:46.936988+00:00 (in 27.399066 seconds) securityworker stdout | 2023-01-17 06:09:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:49 UTC)" (scheduled at 2023-01-17 06:09:19.537472+00:00) securityworker stdout | 2023-01-17 06:09:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:09:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:09:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:09:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:09:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:09:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:09:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:09:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:09:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:51.348605+00:00 (in 29.999543 seconds) gcworker stdout | 2023-01-17 06:09:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:51 UTC)" (scheduled at 2023-01-17 06:09:21.348605+00:00) gcworker stdout | 2023-01-17 06:09:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726161349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:09:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:09:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:09:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:09:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:09:22,060 [231] [DEBUG] [app] Starting request: urn:request:81f8d937-b7cc-4d44-b85b-365cbae81167 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:22,060 [235] [DEBUG] [app] Starting request: urn:request:fce7ad3f-1d34-4a19-89af-6bd96e19630b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:22,064 [243] [DEBUG] [app] Starting request: urn:request:eb652e3e-be8e-4783-b82e-140db55ec3b9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:22,065 [243] [DEBUG] [app] Ending request: urn:request:eb652e3e-be8e-4783-b82e-140db55ec3b9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb652e3e-be8e-4783-b82e-140db55ec3b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:22,065 [230] [DEBUG] [app] Starting request: urn:request:d87da633-ac6e-489e-9d87-957f27e0eb44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:09:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:09:22,065 [230] [DEBUG] [app] Ending request: urn:request:d87da633-ac6e-489e-9d87-957f27e0eb44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d87da633-ac6e-489e-9d87-957f27e0eb44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:09:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:22,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:22,069 [236] [DEBUG] [app] Starting request: urn:request:c29ab15b-3154-46c6-9949-edfc3302fa58 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:22,070 [236] [DEBUG] [app] Ending request: urn:request:c29ab15b-3154-46c6-9949-edfc3302fa58 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c29ab15b-3154-46c6-9949-edfc3302fa58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:22,070 [233] [DEBUG] [app] Starting request: urn:request:c4cca4e9-8f96-4a0e-b431-9385fef6ecba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:09:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:22,070 [233] [DEBUG] [app] Ending request: urn:request:c4cca4e9-8f96-4a0e-b431-9385fef6ecba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4cca4e9-8f96-4a0e-b431-9385fef6ecba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:22,087 [235] [DEBUG] [app] Ending request: urn:request:fce7ad3f-1d34-4a19-89af-6bd96e19630b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fce7ad3f-1d34-4a19-89af-6bd96e19630b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:22,087 [231] [DEBUG] [app] Ending request: urn:request:81f8d937-b7cc-4d44-b85b-365cbae81167 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81f8d937-b7cc-4d44-b85b-365cbae81167', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:22,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:09:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) notificationworker stdout | 2023-01-17 06:09:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:35.646354+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 06:09:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:35 UTC)" (scheduled at 2023-01-17 06:09:25.646354+00:00) notificationworker stdout | 2023-01-17 06:09:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 25, 647255), True, datetime.datetime(2023, 1, 17, 6, 9, 25, 647255), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:35 UTC)" executed successfully builder stdout | 2023-01-17 06:09:28,351 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:28,363 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:28,364 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:28,374 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:09:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:09:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:35.853396+00:00 (in 6.889293 seconds) exportactionlogsworker stdout | 2023-01-17 06:09:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:28 UTC)" (scheduled at 2023-01-17 06:09:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:09:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:09:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:09:30,784 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:09:30,840 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:09:31,594 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:09:31,597 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:09:31,599 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:09:31,601 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:09:31,604 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:09:31,871 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:09:32,224 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:09:32,334 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:09:32,336 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:09:32,799 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:09:35,582 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:09:35,585 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:09:35,587 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:09:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:45.646354+00:00 (in 9.999529 seconds) notificationworker stdout | 2023-01-17 06:09:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:45 UTC)" (scheduled at 2023-01-17 06:09:35.646354+00:00) notificationworker stdout | 2023-01-17 06:09:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 35, 647280), True, datetime.datetime(2023, 1, 17, 6, 9, 35, 647280), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:09:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:09:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:28.963662+00:00 (in 53.109828 seconds) exportactionlogsworker stdout | 2023-01-17 06:09:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:35 UTC)" (scheduled at 2023-01-17 06:09:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:09:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:09:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 35, 854252), True, datetime.datetime(2023, 1, 17, 6, 9, 35, 854252), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:09:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:09:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:09:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:35 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:09:36,022 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-registry stdout | 2023-01-17 06:09:36,058 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:09:36,060 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} expiredappspecifictokenworker stdout | 2023-01-17 06:09:36,064 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-registry stdout | 2023-01-17 06:09:36,066 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:09:36,070 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:09:36,072 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:09:36,074 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:09:36,077 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:09:36,085 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:09:36,087 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:09:37,060 [233] [DEBUG] [app] Starting request: urn:request:028efcaf-872f-4948-b70e-c5f0d8d6d623 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:37,060 [231] [DEBUG] [app] Starting request: urn:request:f74413d2-ff30-4552-9bd3-56a0812d7069 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:37,065 [230] [DEBUG] [app] Starting request: urn:request:a8b42b00-d04b-4959-94c4-3d87ce60baa8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:37,065 [230] [DEBUG] [app] Ending request: urn:request:a8b42b00-d04b-4959-94c4-3d87ce60baa8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a8b42b00-d04b-4959-94c4-3d87ce60baa8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:37,065 [240] [DEBUG] [app] Starting request: urn:request:5f7b03c7-38d4-4e27-a716-79ff459e3aa9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:09:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:09:37,066 [240] [DEBUG] [app] Ending request: urn:request:5f7b03c7-38d4-4e27-a716-79ff459e3aa9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f7b03c7-38d4-4e27-a716-79ff459e3aa9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:37,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:09:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:37,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:37,070 [231] [DEBUG] [app] Starting request: urn:request:b36c3f7c-0854-4cb3-950e-bbb65bac4bc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:37,070 [231] [DEBUG] [app] Ending request: urn:request:b36c3f7c-0854-4cb3-950e-bbb65bac4bc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b36c3f7c-0854-4cb3-950e-bbb65bac4bc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:37,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:37,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:37,072 [235] [DEBUG] [app] Starting request: urn:request:f552fc62-6ad1-4ec0-abb2-23ca8daf0408 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:37,072 [235] [DEBUG] [app] Ending request: urn:request:f552fc62-6ad1-4ec0-abb2-23ca8daf0408 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f552fc62-6ad1-4ec0-abb2-23ca8daf0408', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:37,073 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:37,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:37,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:37,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:37,087 [233] [DEBUG] [app] Ending request: urn:request:028efcaf-872f-4948-b70e-c5f0d8d6d623 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:028efcaf-872f-4948-b70e-c5f0d8d6d623', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:37,087 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:09:37,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:09:37,088 [231] [DEBUG] [app] Ending request: urn:request:f74413d2-ff30-4552-9bd3-56a0812d7069 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f74413d2-ff30-4552-9bd3-56a0812d7069', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:37,089 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 06:09:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 06:09:37,166 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:09:37,171 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} securityscanningnotificationworker stdout | 2023-01-17 06:09:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:09:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:37.537407+00:00 (in 0.280332 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:09:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:37 UTC)" (scheduled at 2023-01-17 06:09:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:09:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:09:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 37, 257442), True, datetime.datetime(2023, 1, 17, 6, 9, 37, 257442), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:09:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:09:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:09:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:09:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:09:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:37.256652+00:00 (in 59.718876 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:09:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:37 UTC)" (scheduled at 2023-01-17 06:09:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:09:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:09:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:09:37,553 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:09:38,060 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:09:38,173 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:09:38,245 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} builder stdout | 2023-01-17 06:09:38,384 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:38,397 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:38,397 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:38,407 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:09:38,502 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:09:39,044 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:09:39,526 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:09:39,664 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:09:39,915 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:09:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:09:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:10.738568+00:00 (in 29.999529 seconds) buildlogsarchiver stdout | 2023-01-17 06:09:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:10 UTC)" (scheduled at 2023-01-17 06:09:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:09:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 9, 40, 739401), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:09:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:09:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:09:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:09:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:09:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:42.941536+00:00 (in 1.792792 seconds) namespacegcworker stdout | 2023-01-17 06:09:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:41 UTC)" (scheduled at 2023-01-17 06:09:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:09:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:09:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 41, 149178), True, datetime.datetime(2023, 1, 17, 6, 9, 41, 149178), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:09:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:09:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:09:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:09:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:09:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:41.148245+00:00 (in 58.206192 seconds) namespacegcworker stdout | 2023-01-17 06:09:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:42 UTC)" (scheduled at 2023-01-17 06:09:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:09:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:09:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:10:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:09:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:55.646354+00:00 (in 9.999558 seconds) notificationworker stdout | 2023-01-17 06:09:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:55 UTC)" (scheduled at 2023-01-17 06:09:45.646354+00:00) notificationworker stdout | 2023-01-17 06:09:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 45, 647179), True, datetime.datetime(2023, 1, 17, 6, 9, 45, 647179), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:09:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:09:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:09:49.537472+00:00 (in 2.600018 seconds) securityworker stdout | 2023-01-17 06:09:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:16 UTC)" (scheduled at 2023-01-17 06:09:46.936988+00:00) securityworker stdout | 2023-01-17 06:09:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:16 UTC)" executed successfully builder stdout | 2023-01-17 06:09:48,417 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:48,430 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:48,430 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:48,441 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:09:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:16.936988+00:00 (in 27.399025 seconds) securityworker stdout | 2023-01-17 06:09:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:19 UTC)" (scheduled at 2023-01-17 06:09:49.537472+00:00) securityworker stdout | 2023-01-17 06:09:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:09:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:09:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:09:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:09:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:09:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:09:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:09:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:09:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:09:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:09:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:09:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:09:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:09:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:09:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:09:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:21.348605+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 06:09:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:21 UTC)" (scheduled at 2023-01-17 06:09:51.348605+00:00) gcworker stdout | 2023-01-17 06:09:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726191349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:09:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:09:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:09:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:09:52,060 [231] [DEBUG] [app] Starting request: urn:request:8d959e45-1bf4-44f0-b48e-9840511b15c3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:52,060 [236] [DEBUG] [app] Starting request: urn:request:2105797a-53ed-4bee-a59c-924f4aff231a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:09:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:09:52,065 [242] [DEBUG] [app] Starting request: urn:request:bdb32774-87b1-4c60-9e37-bf43441b5701 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:52,065 [242] [DEBUG] [app] Ending request: urn:request:bdb32774-87b1-4c60-9e37-bf43441b5701 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bdb32774-87b1-4c60-9e37-bf43441b5701', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:09:52,065 [243] [DEBUG] [app] Starting request: urn:request:6368a2b5-a187-45fb-a8c3-8b33d403a8f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:09:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:09:52,066 [243] [DEBUG] [app] Ending request: urn:request:6368a2b5-a187-45fb-a8c3-8b33d403a8f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6368a2b5-a187-45fb-a8c3-8b33d403a8f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:09:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:09:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:09:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:09:52,071 [235] [DEBUG] [app] Starting request: urn:request:fc9a0437-5354-4ccd-9445-f50837c98a90 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:52,071 [231] [DEBUG] [app] Starting request: urn:request:90ecc6af-55a4-4d02-b02b-891df2b8efaf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:09:52,071 [235] [DEBUG] [app] Ending request: urn:request:fc9a0437-5354-4ccd-9445-f50837c98a90 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc9a0437-5354-4ccd-9445-f50837c98a90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:09:52,071 [231] [DEBUG] [app] Ending request: urn:request:90ecc6af-55a4-4d02-b02b-891df2b8efaf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:90ecc6af-55a4-4d02-b02b-891df2b8efaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:52,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:52,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:09:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:09:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:09:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:09:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:09:52,088 [236] [DEBUG] [app] Ending request: urn:request:2105797a-53ed-4bee-a59c-924f4aff231a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2105797a-53ed-4bee-a59c-924f4aff231a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:52,088 [231] [DEBUG] [app] Ending request: urn:request:8d959e45-1bf4-44f0-b48e-9840511b15c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d959e45-1bf4-44f0-b48e-9840511b15c3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:09:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:52,089 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:09:52,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:09:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:09:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) notificationworker stdout | 2023-01-17 06:09:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:09:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:01.636362+00:00 (in 5.989588 seconds) notificationworker stdout | 2023-01-17 06:09:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:05 UTC)" (scheduled at 2023-01-17 06:09:55.646354+00:00) notificationworker stdout | 2023-01-17 06:09:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:09:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 9, 55, 647161), True, datetime.datetime(2023, 1, 17, 6, 9, 55, 647161), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:09:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:09:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:09:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:05 UTC)" executed successfully builder stdout | 2023-01-17 06:09:58,445 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:09:58,459 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:09:58,459 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:09:58,470 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:00,816 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:10:00,870 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:10:01,602 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:10:01,604 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:10:01,607 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:10:01,610 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:10:01,612 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:10:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:05.646354+00:00 (in 4.009582 seconds) notificationworker stdout | 2023-01-17 06:10:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:01 UTC)" (scheduled at 2023-01-17 06:10:01.636362+00:00) notificationworker stdout | 2023-01-17 06:10:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:10:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:01 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:10:01,906 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:10:02,253 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:10:02,355 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:10:02,369 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:10:02,833 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:10:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:10:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:19.346120+00:00 (in 14.408883 seconds) repositorygcworker stdout | 2023-01-17 06:10:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:04 UTC)" (scheduled at 2023-01-17 06:10:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:10:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:10:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 4, 937609), True, datetime.datetime(2023, 1, 17, 6, 10, 4, 937609), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:10:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:10:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:10:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:10:05,589 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:10:05,592 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:10:05,594 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:10:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:15.646354+00:00 (in 9.999574 seconds) notificationworker stdout | 2023-01-17 06:10:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:15 UTC)" (scheduled at 2023-01-17 06:10:05.646354+00:00) notificationworker stdout | 2023-01-17 06:10:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 5, 647159), True, datetime.datetime(2023, 1, 17, 6, 10, 5, 647159), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:15 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:10:06,057 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-registry stdout | 2023-01-17 06:10:06,067 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:10:06,070 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:10:06,073 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:10:06,078 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:10:06,080 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:10:06,084 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:10:06,087 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:10:06,094 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:10:06,096 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} expiredappspecifictokenworker stdout | 2023-01-17 06:10:06,099 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:10:07,060 [233] [DEBUG] [app] Starting request: urn:request:a57472cf-971c-4b44-8503-e293a0a65119 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:07,060 [235] [DEBUG] [app] Starting request: urn:request:9c74aa61-d0d9-4dd2-9ba5-a32b131cf66a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:07,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:07,065 [230] [DEBUG] [app] Starting request: urn:request:d2edefea-e28a-4d52-9928-370849e4d569 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:07,065 [241] [DEBUG] [app] Starting request: urn:request:49b8e7e5-fb01-46ad-a4ae-2f73a2a249e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:07,066 [230] [DEBUG] [app] Ending request: urn:request:d2edefea-e28a-4d52-9928-370849e4d569 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2edefea-e28a-4d52-9928-370849e4d569', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:07,066 [241] [DEBUG] [app] Ending request: urn:request:49b8e7e5-fb01-46ad-a4ae-2f73a2a249e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49b8e7e5-fb01-46ad-a4ae-2f73a2a249e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:10:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:10:07,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:07,071 [236] [DEBUG] [app] Starting request: urn:request:9eff35e3-06a2-4076-9bd4-490809ef5f22 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:07,071 [231] [DEBUG] [app] Starting request: urn:request:240be81a-9191-4617-8d71-eaf362b90ab2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:07,071 [236] [DEBUG] [app] Ending request: urn:request:9eff35e3-06a2-4076-9bd4-490809ef5f22 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9eff35e3-06a2-4076-9bd4-490809ef5f22', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:07,071 [231] [DEBUG] [app] Ending request: urn:request:240be81a-9191-4617-8d71-eaf362b90ab2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:240be81a-9191-4617-8d71-eaf362b90ab2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:07,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:07,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:07,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:07,088 [235] [DEBUG] [app] Ending request: urn:request:9c74aa61-d0d9-4dd2-9ba5-a32b131cf66a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c74aa61-d0d9-4dd2-9ba5-a32b131cf66a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:07,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:07,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:10:07,090 [233] [DEBUG] [app] Ending request: urn:request:a57472cf-971c-4b44-8503-e293a0a65119 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a57472cf-971c-4b44-8503-e293a0a65119', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:07,090 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:07,091 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.031) storagereplication stdout | 2023-01-17 06:10:07,193 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:10:07,205 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:10:07,588 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:10:08,078 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:10:08,178 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:10:08,280 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} builder stdout | 2023-01-17 06:10:08,481 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:08,494 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:08,494 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:08,504 [60] [DEBUG] [data.database] Disconnecting from database. repositoryactioncounter stdout | 2023-01-17 06:10:08,531 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:10:09,080 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:10:09,543 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:10:09,699 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:10:09,921 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:10:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:10:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:40.738568+00:00 (in 29.999555 seconds) buildlogsarchiver stdout | 2023-01-17 06:10:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:40 UTC)" (scheduled at 2023-01-17 06:10:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:10:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 10, 10, 739401), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:10:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:10:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:10:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:10:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:25.646354+00:00 (in 9.999571 seconds) notificationworker stdout | 2023-01-17 06:10:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:25 UTC)" (scheduled at 2023-01-17 06:10:15.646354+00:00) notificationworker stdout | 2023-01-17 06:10:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 15, 647188), True, datetime.datetime(2023, 1, 17, 6, 10, 15, 647188), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:10:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:19.537472+00:00 (in 2.599991 seconds) securityworker stdout | 2023-01-17 06:10:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:46 UTC)" (scheduled at 2023-01-17 06:10:16.936988+00:00) securityworker stdout | 2023-01-17 06:10:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:16,962 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:16,965 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:16,965 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:46 UTC)" executed successfully builder stdout | 2023-01-17 06:10:18,514 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:18,527 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:18,527 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:18,537 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:10:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:10:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:04.936835+00:00 (in 45.590291 seconds) repositorygcworker stdout | 2023-01-17 06:10:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:19 UTC)" (scheduled at 2023-01-17 06:10:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:10:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:10:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:10:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:46.936988+00:00 (in 27.399013 seconds) securityworker stdout | 2023-01-17 06:10:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:49 UTC)" (scheduled at 2023-01-17 06:10:19.537472+00:00) securityworker stdout | 2023-01-17 06:10:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:10:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:10:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:19,561 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:19,561 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:10:19,561 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:10:19,561 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:10:19,561 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:19,562 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:19,562 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:19,562 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:10:19,562 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:10:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:10:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:51.348605+00:00 (in 29.999517 seconds) gcworker stdout | 2023-01-17 06:10:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:51 UTC)" (scheduled at 2023-01-17 06:10:21.348605+00:00) gcworker stdout | 2023-01-17 06:10:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726221349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:10:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:10:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:10:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:10:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:10:22,060 [235] [DEBUG] [app] Starting request: urn:request:6cd920af-1876-40fb-8f89-66b485421e53 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:22,060 [231] [DEBUG] [app] Starting request: urn:request:83c81b21-7695-4a2b-ab6a-8710d1ca3d1a (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:22,064 [243] [DEBUG] [app] Starting request: urn:request:a01207aa-7bf7-4355-93a5-e394de3b6397 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:22,065 [241] [DEBUG] [app] Starting request: urn:request:b9ed71dd-7807-485a-82f1-27e09fcf7cf2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:22,065 [243] [DEBUG] [app] Ending request: urn:request:a01207aa-7bf7-4355-93a5-e394de3b6397 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a01207aa-7bf7-4355-93a5-e394de3b6397', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:10:22,065 [241] [DEBUG] [app] Ending request: urn:request:b9ed71dd-7807-485a-82f1-27e09fcf7cf2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9ed71dd-7807-485a-82f1-27e09fcf7cf2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:22,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:10:22,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:10:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:22,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:22,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:22,069 [236] [DEBUG] [app] Starting request: urn:request:a2b96182-1e6f-47ca-8586-4860202a3e72 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:22,070 [233] [DEBUG] [app] Starting request: urn:request:45f8199b-392a-4508-90ed-4f08d8fbd844 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:22,070 [236] [DEBUG] [app] Ending request: urn:request:a2b96182-1e6f-47ca-8586-4860202a3e72 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a2b96182-1e6f-47ca-8586-4860202a3e72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:10:22,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:22,070 [233] [DEBUG] [app] Ending request: urn:request:45f8199b-392a-4508-90ed-4f08d8fbd844 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45f8199b-392a-4508-90ed-4f08d8fbd844', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:22,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:22,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:22,087 [235] [DEBUG] [app] Ending request: urn:request:6cd920af-1876-40fb-8f89-66b485421e53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cd920af-1876-40fb-8f89-66b485421e53', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:22,087 [231] [DEBUG] [app] Ending request: urn:request:83c81b21-7695-4a2b-ab6a-8710d1ca3d1a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83c81b21-7695-4a2b-ab6a-8710d1ca3d1a', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:22,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:22,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:10:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) notificationworker stdout | 2023-01-17 06:10:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:35.646354+00:00 (in 9.999524 seconds) notificationworker stdout | 2023-01-17 06:10:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:35 UTC)" (scheduled at 2023-01-17 06:10:25.646354+00:00) notificationworker stdout | 2023-01-17 06:10:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 25, 647250), True, datetime.datetime(2023, 1, 17, 6, 10, 25, 647250), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:35 UTC)" executed successfully builder stdout | 2023-01-17 06:10:28,547 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:28,560 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:28,560 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:28,571 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:10:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:10:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:35.853396+00:00 (in 6.889237 seconds) exportactionlogsworker stdout | 2023-01-17 06:10:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:28 UTC)" (scheduled at 2023-01-17 06:10:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:10:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:10:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:28 UTC)" executed successfully servicekey stdout | 2023-01-17 06:10:30,442 [88] [DEBUG] [apscheduler.scheduler] Looking for jobs to run servicekey stdout | 2023-01-17 06:10:30,442 [88] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 07:05:30.441589+00:00 (in 3299.998987 seconds) servicekey stdout | 2023-01-17 06:10:30,442 [88] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 07:05:30 UTC)" (scheduled at 2023-01-17 06:10:30.441589+00:00) servicekey stdout | 2023-01-17 06:10:30,443 [88] [DEBUG] [__main__] Starting automatic refresh of service key HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw to new expiration 2023-01-17 08:10:30.442992 servicekey stdout | 2023-01-17 06:10:30,443 [88] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw', datetime.datetime(2023, 1, 10, 6, 10, 30, 443215), None, 1, 0]) servicekey stdout | 2023-01-17 06:10:30,454 [88] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com:443', 'HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw', 'quay', '{"n": "vyADstWkSWTJPDl7kqlPmO1eZbt9FNsmZ9_-tbg63McWVJDNdbRHLiI9N4_ZE8_B88DplENLF1WuADJhqVorEQVmjdVIBzD4kdVe4iOvR2mSA2eHlmDWzF5viMYFkKU_w2mHvCckDtnOpJaGnE-hozsrw625V5rDvMrC481O-6vcYOwDb5g3pk-lMMI2aAKURo-kS5osmLh73vHiGKqLN2ymMMV0zZHsa1jXfBLBMpRwtApJgBqdgA_87e3QPLGlOzwGv24FVCLECm-L_XxSQAGndNVnVCYS43gxZvjhqML1fPeCA3dD91GmGYsuwW7MMlYcvefd_8JwHXJO9eadlw", "e": "AQAB", "kty": "RSA", "kid": "HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw"}', '{"created_by": "CLI tool"}', datetime.datetime(2023, 1, 17, 4, 49, 11, 724988), datetime.datetime(2023, 1, 17, 8, 10, 30, 442992), None, 5, 5]) servicekey stdout | 2023-01-17 06:10:30,460 [88] [DEBUG] [__main__] Finished automatic refresh of service key HP779Q52cppMdbd7_fMO2fiURX6C0iGKIxIBTIi_lFw with new expiration 2023-01-17 08:10:30.442992 servicekey stdout | 2023-01-17 06:10:30,460 [88] [DEBUG] [data.database] Disconnecting from database. servicekey stdout | 2023-01-17 06:10:30,460 [88] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2023-01-17 07:05:30 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:10:30,852 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:10:30,886 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:10:31,610 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:10:31,612 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:10:31,615 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:10:31,617 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:10:31,619 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:10:31,941 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:10:32,287 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:10:32,375 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:10:32,390 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:10:32,844 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:10:35,598 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:10:35,600 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:10:35,603 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:10:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:45.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 06:10:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:45 UTC)" (scheduled at 2023-01-17 06:10:35.646354+00:00) notificationworker stdout | 2023-01-17 06:10:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 35, 647198), True, datetime.datetime(2023, 1, 17, 6, 10, 35, 647198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:10:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:10:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:28.963662+00:00 (in 53.109792 seconds) exportactionlogsworker stdout | 2023-01-17 06:10:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:35 UTC)" (scheduled at 2023-01-17 06:10:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:10:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:10:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 35, 854242), True, datetime.datetime(2023, 1, 17, 6, 10, 35, 854242), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:10:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:10:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:10:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:10:36,076 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:10:36,079 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:10:36,081 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:10:36,087 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:10:36,089 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} buildlogsarchiver stdout | 2023-01-17 06:10:36,091 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-registry stdout | 2023-01-17 06:10:36,094 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:10:36,096 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:10:36,101 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:10:36,105 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} expiredappspecifictokenworker stdout | 2023-01-17 06:10:36,120 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:10:37,059 [235] [DEBUG] [app] Starting request: urn:request:8e3ff344-68e6-4ae6-9b8b-5669e32cfc20 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:37,059 [231] [DEBUG] [app] Starting request: urn:request:5b1cab89-bb2f-4979-b078-6f32146a32cd (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:37,065 [230] [DEBUG] [app] Starting request: urn:request:a7549c93-dbba-4a44-9759-7d49f414ab42 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:37,065 [241] [DEBUG] [app] Starting request: urn:request:ef29e806-88c7-4f47-86d7-921c8977f919 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:37,065 [230] [DEBUG] [app] Ending request: urn:request:a7549c93-dbba-4a44-9759-7d49f414ab42 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a7549c93-dbba-4a44-9759-7d49f414ab42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:37,065 [241] [DEBUG] [app] Ending request: urn:request:ef29e806-88c7-4f47-86d7-921c8977f919 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef29e806-88c7-4f47-86d7-921c8977f919', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:37,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:10:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:10:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:37,070 [236] [DEBUG] [app] Starting request: urn:request:b9959df4-d2b7-4d8e-971c-8664e67d83ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:37,070 [236] [DEBUG] [app] Ending request: urn:request:b9959df4-d2b7-4d8e-971c-8664e67d83ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9959df4-d2b7-4d8e-971c-8664e67d83ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:37,070 [233] [DEBUG] [app] Starting request: urn:request:5f37ee1f-e29c-4cfc-97d4-6a4fb65e9e2e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:37,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:37,070 [233] [DEBUG] [app] Ending request: urn:request:5f37ee1f-e29c-4cfc-97d4-6a4fb65e9e2e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f37ee1f-e29c-4cfc-97d4-6a4fb65e9e2e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:37,085 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:37,088 [235] [DEBUG] [app] Ending request: urn:request:8e3ff344-68e6-4ae6-9b8b-5669e32cfc20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8e3ff344-68e6-4ae6-9b8b-5669e32cfc20', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:37,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) gunicorn-web stdout | 2023-01-17 06:10:37,094 [231] [DEBUG] [app] Ending request: urn:request:5b1cab89-bb2f-4979-b078-6f32146a32cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5b1cab89-bb2f-4979-b078-6f32146a32cd', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:37,094 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.035 47 0.036) gunicorn-web stdout | 2023-01-17 06:10:37,094 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 06:10:37,224 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:10:37,227 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} securityscanningnotificationworker stdout | 2023-01-17 06:10:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:10:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:37.537407+00:00 (in 0.280370 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:10:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:37 UTC)" (scheduled at 2023-01-17 06:10:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:10:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:10:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 37, 257423), True, datetime.datetime(2023, 1, 17, 6, 10, 37, 257423), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:10:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:10:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:10:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:10:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:10:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:23.537048+00:00 (in 45.999231 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:10:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:37 UTC)" (scheduled at 2023-01-17 06:10:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:10:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:10:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:10:37,624 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:10:38,088 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:10:38,198 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:10:38,300 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:10:38,566 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 06:10:38,581 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:38,594 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:38,594 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:38,604 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:10:39,087 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:10:39,548 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:10:39,734 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:10:39,956 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:10:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:10:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:10.738568+00:00 (in 29.999547 seconds) buildlogsarchiver stdout | 2023-01-17 06:10:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:10 UTC)" (scheduled at 2023-01-17 06:10:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:10:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 10, 40, 739418), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:10:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:10:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:10:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:10:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:10:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:42.941536+00:00 (in 1.792747 seconds) namespacegcworker stdout | 2023-01-17 06:10:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:41 UTC)" (scheduled at 2023-01-17 06:10:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:10:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:10:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 41, 149261), True, datetime.datetime(2023, 1, 17, 6, 10, 41, 149261), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:10:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:10:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:10:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:10:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:10:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:41.148245+00:00 (in 58.206158 seconds) namespacegcworker stdout | 2023-01-17 06:10:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:42 UTC)" (scheduled at 2023-01-17 06:10:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:10:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:10:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:11:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:10:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:55.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 06:10:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:55 UTC)" (scheduled at 2023-01-17 06:10:45.646354+00:00) notificationworker stdout | 2023-01-17 06:10:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 45, 647218), True, datetime.datetime(2023, 1, 17, 6, 10, 45, 647218), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:10:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:10:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:10:49.537472+00:00 (in 2.600035 seconds) securityworker stdout | 2023-01-17 06:10:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:16 UTC)" (scheduled at 2023-01-17 06:10:46.936988+00:00) securityworker stdout | 2023-01-17 06:10:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:16 UTC)" executed successfully builder stdout | 2023-01-17 06:10:48,607 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:48,620 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:48,620 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:48,631 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:10:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:16.936988+00:00 (in 27.399052 seconds) securityworker stdout | 2023-01-17 06:10:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:19 UTC)" (scheduled at 2023-01-17 06:10:49.537472+00:00) securityworker stdout | 2023-01-17 06:10:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:10:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:10:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:10:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:10:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:10:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:10:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:10:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:10:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:10:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:10:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:10:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:10:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:10:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:10:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:10:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:21.348605+00:00 (in 29.999499 seconds) gcworker stdout | 2023-01-17 06:10:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:21 UTC)" (scheduled at 2023-01-17 06:10:51.348605+00:00) gcworker stdout | 2023-01-17 06:10:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726251349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:10:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:10:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:10:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:10:52,060 [235] [DEBUG] [app] Starting request: urn:request:c8625f57-099b-4f60-a6fb-93e20d6fa962 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:52,060 [233] [DEBUG] [app] Starting request: urn:request:2f080a5d-4cf8-4057-b022-321be4146cbe (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:10:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:52,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:10:52,064 [242] [DEBUG] [app] Starting request: urn:request:e7810066-9a12-4b04-83b7-0eb919d1cd57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:52,065 [230] [DEBUG] [app] Starting request: urn:request:5f8d80c1-abde-488f-af7c-8fb7f93fc036 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:10:52,065 [242] [DEBUG] [app] Ending request: urn:request:e7810066-9a12-4b04-83b7-0eb919d1cd57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e7810066-9a12-4b04-83b7-0eb919d1cd57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:10:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:10:52,065 [230] [DEBUG] [app] Ending request: urn:request:5f8d80c1-abde-488f-af7c-8fb7f93fc036 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f8d80c1-abde-488f-af7c-8fb7f93fc036', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:52,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:10:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:10:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:10:52,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:10:52,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:52,069 [233] [DEBUG] [app] Starting request: urn:request:304b33a1-eec7-40d4-8390-389d49aa82b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:52,069 [233] [DEBUG] [app] Ending request: urn:request:304b33a1-eec7-40d4-8390-389d49aa82b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:304b33a1-eec7-40d4-8390-389d49aa82b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:52,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:52,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:10:52,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:52,071 [236] [DEBUG] [app] Starting request: urn:request:bc85ecdd-542e-4cae-a3a9-8ad4090503d3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:10:52,071 [236] [DEBUG] [app] Ending request: urn:request:bc85ecdd-542e-4cae-a3a9-8ad4090503d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc85ecdd-542e-4cae-a3a9-8ad4090503d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:10:52,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:10:52,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:10:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:10:52,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:10:52,087 [235] [DEBUG] [app] Ending request: urn:request:c8625f57-099b-4f60-a6fb-93e20d6fa962 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c8625f57-099b-4f60-a6fb-93e20d6fa962', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:10:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:52,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:10:52,088 [233] [DEBUG] [app] Ending request: urn:request:2f080a5d-4cf8-4057-b022-321be4146cbe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f080a5d-4cf8-4057-b022-321be4146cbe', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:10:52,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:10:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:10:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:10:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:10:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:01.636362+00:00 (in 5.989538 seconds) notificationworker stdout | 2023-01-17 06:10:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:05 UTC)" (scheduled at 2023-01-17 06:10:55.646354+00:00) notificationworker stdout | 2023-01-17 06:10:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:10:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 10, 55, 647222), True, datetime.datetime(2023, 1, 17, 6, 10, 55, 647222), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:10:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:10:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:10:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:05 UTC)" executed successfully builder stdout | 2023-01-17 06:10:58,641 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:10:58,654 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:10:58,655 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:10:58,666 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:00,881 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:11:00,918 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:11:01,618 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:11:01,620 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:11:01,623 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:11:01,625 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:11:01,627 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} notificationworker stdout | 2023-01-17 06:11:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:05.646354+00:00 (in 4.009585 seconds) notificationworker stdout | 2023-01-17 06:11:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:01 UTC)" (scheduled at 2023-01-17 06:11:01.636362+00:00) notificationworker stdout | 2023-01-17 06:11:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:11:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:01 UTC)" executed successfully queuecleanupworker stdout | 2023-01-17 06:11:01,953 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:11:02,311 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:11:02,410 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:11:02,424 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:11:02,880 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:11:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:11:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:17.345601+00:00 (in 12.408318 seconds) repositorygcworker stdout | 2023-01-17 06:11:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:04 UTC)" (scheduled at 2023-01-17 06:11:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:11:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:11:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 4, 937692), True, datetime.datetime(2023, 1, 17, 6, 11, 4, 937692), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:11:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:11:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:11:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:11:05,605 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:11:05,608 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:11:05,611 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:11:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:15.646354+00:00 (in 9.999566 seconds) notificationworker stdout | 2023-01-17 06:11:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:15 UTC)" (scheduled at 2023-01-17 06:11:05.646354+00:00) notificationworker stdout | 2023-01-17 06:11:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 5, 647194), True, datetime.datetime(2023, 1, 17, 6, 11, 5, 647194), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:11:06,085 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:11:06,088 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:11:06,092 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:11:06,097 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:11:06,099 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:11:06,102 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:11:06,106 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:11:06,109 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:11:06,113 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:11:06,121 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:11:06,139 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:11:07,060 [236] [DEBUG] [app] Starting request: urn:request:3a03d6c1-2aff-46c6-b109-82a3a6a6fb2d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:07,060 [231] [DEBUG] [app] Starting request: urn:request:943a1f9a-2da3-42a1-b428-eb660e062dc8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:07,065 [241] [DEBUG] [app] Starting request: urn:request:56caf63c-3256-458c-9486-25283a23c693 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:07,065 [243] [DEBUG] [app] Starting request: urn:request:f6a393a6-c70e-4607-b83b-bc7cbbb4184e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:07,065 [241] [DEBUG] [app] Ending request: urn:request:56caf63c-3256-458c-9486-25283a23c693 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56caf63c-3256-458c-9486-25283a23c693', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:11:07,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:11:07,065 [243] [DEBUG] [app] Ending request: urn:request:f6a393a6-c70e-4607-b83b-bc7cbbb4184e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f6a393a6-c70e-4607-b83b-bc7cbbb4184e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:11:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:11:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:07,070 [233] [DEBUG] [app] Starting request: urn:request:6f7cc708-a5d2-4ae7-a2da-c2b69cb3fa0d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:07,070 [233] [DEBUG] [app] Ending request: urn:request:6f7cc708-a5d2-4ae7-a2da-c2b69cb3fa0d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f7cc708-a5d2-4ae7-a2da-c2b69cb3fa0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:07,070 [231] [DEBUG] [app] Starting request: urn:request:6d495f73-014d-4822-a612-d4f45f2af23b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:07,071 [231] [DEBUG] [app] Ending request: urn:request:6d495f73-014d-4822-a612-d4f45f2af23b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d495f73-014d-4822-a612-d4f45f2af23b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:07,088 [231] [DEBUG] [app] Ending request: urn:request:943a1f9a-2da3-42a1-b428-eb660e062dc8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:943a1f9a-2da3-42a1-b428-eb660e062dc8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:07,088 [236] [DEBUG] [app] Ending request: urn:request:3a03d6c1-2aff-46c6-b109-82a3a6a6fb2d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a03d6c1-2aff-46c6-b109-82a3a6a6fb2d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:11:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) storagereplication stdout | 2023-01-17 06:11:07,239 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:11:07,261 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:11:07,637 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:11:08,116 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:11:08,205 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:11:08,308 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:11:08,601 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 06:11:08,673 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:08,685 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:08,685 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:08,696 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:11:09,094 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:11:09,584 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:11:09,768 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:11:09,992 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:11:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:11:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:40.738568+00:00 (in 29.999488 seconds) buildlogsarchiver stdout | 2023-01-17 06:11:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:40 UTC)" (scheduled at 2023-01-17 06:11:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:11:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 11, 10, 739517), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:11:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:11:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:11:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:25.646354+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 06:11:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:25 UTC)" (scheduled at 2023-01-17 06:11:15.646354+00:00) notificationworker stdout | 2023-01-17 06:11:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 15, 647242), True, datetime.datetime(2023, 1, 17, 6, 11, 15, 647242), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:11:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:19.537472+00:00 (in 2.600040 seconds) securityworker stdout | 2023-01-17 06:11:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:46 UTC)" (scheduled at 2023-01-17 06:11:16.936988+00:00) securityworker stdout | 2023-01-17 06:11:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:11:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:11:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:19.346120+00:00 (in 2.000054 seconds) repositorygcworker stdout | 2023-01-17 06:11:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:17 UTC)" (scheduled at 2023-01-17 06:11:17.345601+00:00) repositorygcworker stdout | 2023-01-17 06:11:17,346 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 11, 17, 346420), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:11:17,356 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 17, 346420), True, datetime.datetime(2023, 1, 17, 6, 11, 17, 346420), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:11:17,360 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 17, 346420), True, datetime.datetime(2023, 1, 17, 6, 11, 17, 346420), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 11, 17, 346420), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:11:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:11:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:17 UTC)" executed successfully builder stdout | 2023-01-17 06:11:18,706 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:18,719 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:18,719 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:18,729 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:11:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:11:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:04.936835+00:00 (in 45.590286 seconds) repositorygcworker stdout | 2023-01-17 06:11:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:19 UTC)" (scheduled at 2023-01-17 06:11:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:11:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:11:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:11:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:46.936988+00:00 (in 27.399096 seconds) securityworker stdout | 2023-01-17 06:11:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:49 UTC)" (scheduled at 2023-01-17 06:11:19.537472+00:00) securityworker stdout | 2023-01-17 06:11:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:11:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:11:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:19,540 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:19,540 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:11:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:11:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:11:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:19,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:19,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:19,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:11:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:11:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:11:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:51.348605+00:00 (in 29.999540 seconds) gcworker stdout | 2023-01-17 06:11:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:51 UTC)" (scheduled at 2023-01-17 06:11:21.348605+00:00) gcworker stdout | 2023-01-17 06:11:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726281349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:11:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:11:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:11:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:11:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:11:22,059 [231] [DEBUG] [app] Starting request: urn:request:603dd739-bb86-454e-a1d5-723782ff7c8d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:22,059 [233] [DEBUG] [app] Starting request: urn:request:1fb79cdb-cf13-4bb3-89e9-d80886dd4971 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:22,065 [230] [DEBUG] [app] Starting request: urn:request:b1847238-b8f8-4402-8d69-656d87599576 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:22,065 [243] [DEBUG] [app] Starting request: urn:request:934340d5-4d02-451e-bdd0-b28cd8f1de36 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:22,065 [230] [DEBUG] [app] Ending request: urn:request:b1847238-b8f8-4402-8d69-656d87599576 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1847238-b8f8-4402-8d69-656d87599576', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:11:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:11:22,065 [243] [DEBUG] [app] Ending request: urn:request:934340d5-4d02-451e-bdd0-b28cd8f1de36 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:934340d5-4d02-451e-bdd0-b28cd8f1de36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:11:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:22,070 [235] [DEBUG] [app] Starting request: urn:request:a1312d72-fd92-48c1-abc1-e24ca153f2e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:22,071 [236] [DEBUG] [app] Starting request: urn:request:a69b5e99-3df2-4457-9c04-cced35271d64 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:22,071 [235] [DEBUG] [app] Ending request: urn:request:a1312d72-fd92-48c1-abc1-e24ca153f2e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1312d72-fd92-48c1-abc1-e24ca153f2e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:11:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:22,071 [236] [DEBUG] [app] Ending request: urn:request:a69b5e99-3df2-4457-9c04-cced35271d64 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a69b5e99-3df2-4457-9c04-cced35271d64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:22,088 [231] [DEBUG] [app] Ending request: urn:request:603dd739-bb86-454e-a1d5-723782ff7c8d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:603dd739-bb86-454e-a1d5-723782ff7c8d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:22,088 [233] [DEBUG] [app] Ending request: urn:request:1fb79cdb-cf13-4bb3-89e9-d80886dd4971 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1fb79cdb-cf13-4bb3-89e9-d80886dd4971', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 06:11:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:11:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:11:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:37.256652+00:00 (in 13.719187 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:11:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:23 UTC)" (scheduled at 2023-01-17 06:11:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:11:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 11, 23, 537796), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:11:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 23, 537796), True, datetime.datetime(2023, 1, 17, 6, 11, 23, 537796), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:11:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 23, 537796), True, datetime.datetime(2023, 1, 17, 6, 11, 23, 537796), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 11, 23, 537796), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:11:23,556 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:11:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:35.646354+00:00 (in 9.999513 seconds) notificationworker stdout | 2023-01-17 06:11:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:35 UTC)" (scheduled at 2023-01-17 06:11:25.646354+00:00) notificationworker stdout | 2023-01-17 06:11:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 25, 647219), True, datetime.datetime(2023, 1, 17, 6, 11, 25, 647219), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:35 UTC)" executed successfully builder stdout | 2023-01-17 06:11:28,740 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:28,752 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:28,752 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:28,763 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:11:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:11:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:35.853396+00:00 (in 6.889301 seconds) exportactionlogsworker stdout | 2023-01-17 06:11:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:28 UTC)" (scheduled at 2023-01-17 06:11:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:11:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:11:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:30,903 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:11:30,953 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:11:31,625 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:11:31,628 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:11:31,630 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:11:31,633 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:11:31,635 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:11:31,962 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:11:32,339 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:11:32,438 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:11:32,440 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:11:32,897 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:11:35,613 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:11:35,615 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:11:35,617 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:11:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:45.646354+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 06:11:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:45 UTC)" (scheduled at 2023-01-17 06:11:35.646354+00:00) notificationworker stdout | 2023-01-17 06:11:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 35, 647194), True, datetime.datetime(2023, 1, 17, 6, 11, 35, 647194), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:35,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:11:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:11:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:54.963413+00:00 (in 19.109605 seconds) exportactionlogsworker stdout | 2023-01-17 06:11:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:35 UTC)" (scheduled at 2023-01-17 06:11:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:11:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:11:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 35, 854198), True, datetime.datetime(2023, 1, 17, 6, 11, 35, 854198), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:11:35,864 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:11:35,864 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:11:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:11:36,092 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:11:36,097 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:11:36,101 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:11:36,104 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:11:36,109 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:11:36,111 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:11:36,113 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:11:36,116 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:11:36,120 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:11:36,126 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:11:36,174 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:11:37,060 [231] [DEBUG] [app] Starting request: urn:request:50565f7f-b460-47e4-a20b-867f60b3f056 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:37,060 [236] [DEBUG] [app] Starting request: urn:request:02a5f981-f119-490f-a764-5aa6aee15ac0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:37,065 [230] [DEBUG] [app] Starting request: urn:request:be05a15e-ce1d-4fb4-8294-09ec7f97f60a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:37,065 [230] [DEBUG] [app] Ending request: urn:request:be05a15e-ce1d-4fb4-8294-09ec7f97f60a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be05a15e-ce1d-4fb4-8294-09ec7f97f60a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:11:37,065 [241] [DEBUG] [app] Starting request: urn:request:ab7c532a-bb8d-43ce-b0f0-a058bde01446 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:11:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:11:37,066 [241] [DEBUG] [app] Ending request: urn:request:ab7c532a-bb8d-43ce-b0f0-a058bde01446 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab7c532a-bb8d-43ce-b0f0-a058bde01446', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:11:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:11:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:37,070 [231] [DEBUG] [app] Starting request: urn:request:e538e588-7fc8-472a-a122-ec996b4a1d35 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:37,071 [231] [DEBUG] [app] Ending request: urn:request:e538e588-7fc8-472a-a122-ec996b4a1d35 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e538e588-7fc8-472a-a122-ec996b4a1d35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:37,071 [235] [DEBUG] [app] Starting request: urn:request:765f8f60-1e7b-48d9-bc7f-ec3a78fff486 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:37,072 [235] [DEBUG] [app] Ending request: urn:request:765f8f60-1e7b-48d9-bc7f-ec3a78fff486 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:765f8f60-1e7b-48d9-bc7f-ec3a78fff486', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:37,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:11:37,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:37,087 [231] [DEBUG] [app] Ending request: urn:request:50565f7f-b460-47e4-a20b-867f60b3f056 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:50565f7f-b460-47e4-a20b-867f60b3f056', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:11:37,088 [236] [DEBUG] [app] Ending request: urn:request:02a5f981-f119-490f-a764-5aa6aee15ac0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:02a5f981-f119-490f-a764-5aa6aee15ac0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 06:11:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:11:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:11:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:37.537407+00:00 (in 0.280327 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:11:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:37 UTC)" (scheduled at 2023-01-17 06:11:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:11:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:11:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 37, 257505), True, datetime.datetime(2023, 1, 17, 6, 11, 37, 257505), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:11:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:11:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:11:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:11:37,274 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:11:37,297 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} securityscanningnotificationworker stdout | 2023-01-17 06:11:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:11:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:37.256652+00:00 (in 59.718816 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:11:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:37 UTC)" (scheduled at 2023-01-17 06:11:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:11:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:11:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:11:37,673 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:11:38,135 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:11:38,229 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:11:38,327 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:11:38,636 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 06:11:38,773 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:38,785 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:38,785 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:38,795 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:11:39,131 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:11:39,615 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:11:39,798 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:11:40,026 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:11:40,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:11:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:10.738568+00:00 (in 29.999419 seconds) buildlogsarchiver stdout | 2023-01-17 06:11:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:10 UTC)" (scheduled at 2023-01-17 06:11:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:11:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 11, 40, 739579), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:11:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:11:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:11:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:11:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:11:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:42.941536+00:00 (in 1.792704 seconds) namespacegcworker stdout | 2023-01-17 06:11:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:41 UTC)" (scheduled at 2023-01-17 06:11:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:11:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:11:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 41, 149260), True, datetime.datetime(2023, 1, 17, 6, 11, 41, 149260), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:11:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:11:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:11:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:11:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:11:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:06.941185+00:00 (in 23.999181 seconds) namespacegcworker stdout | 2023-01-17 06:11:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:42 UTC)" (scheduled at 2023-01-17 06:11:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:11:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:11:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:12:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:55.646354+00:00 (in 9.999535 seconds) notificationworker stdout | 2023-01-17 06:11:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:55 UTC)" (scheduled at 2023-01-17 06:11:45.646354+00:00) notificationworker stdout | 2023-01-17 06:11:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 45, 647183), True, datetime.datetime(2023, 1, 17, 6, 11, 45, 647183), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:11:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:11:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:11:49.537472+00:00 (in 2.600059 seconds) securityworker stdout | 2023-01-17 06:11:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:16 UTC)" (scheduled at 2023-01-17 06:11:46.936988+00:00) securityworker stdout | 2023-01-17 06:11:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:16 UTC)" executed successfully builder stdout | 2023-01-17 06:11:48,805 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:48,817 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:48,817 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:48,828 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:11:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:16.936988+00:00 (in 27.399051 seconds) securityworker stdout | 2023-01-17 06:11:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:19 UTC)" (scheduled at 2023-01-17 06:11:49.537472+00:00) securityworker stdout | 2023-01-17 06:11:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:11:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:11:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:11:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:11:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:11:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:11:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:11:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:11:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:11:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:11:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:11:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:11:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:11:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:11:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:11:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:21.348605+00:00 (in 29.999487 seconds) gcworker stdout | 2023-01-17 06:11:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:21 UTC)" (scheduled at 2023-01-17 06:11:51.348605+00:00) gcworker stdout | 2023-01-17 06:11:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726311349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:11:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:11:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:11:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:11:52,060 [236] [DEBUG] [app] Starting request: urn:request:e6f0dd7e-7efe-4dda-a500-b3e07723ecd9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:52,060 [231] [DEBUG] [app] Starting request: urn:request:3f987d09-7d0a-40fc-8800-386492544cea (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:11:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:11:52,065 [230] [DEBUG] [app] Starting request: urn:request:36eb0df6-79dd-4214-aed6-da7372e14312 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:52,066 [242] [DEBUG] [app] Starting request: urn:request:4d620a6a-a5b4-46df-abf1-307f5c7ae8ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:11:52,066 [230] [DEBUG] [app] Ending request: urn:request:36eb0df6-79dd-4214-aed6-da7372e14312 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36eb0df6-79dd-4214-aed6-da7372e14312', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:11:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:11:52,066 [242] [DEBUG] [app] Ending request: urn:request:4d620a6a-a5b4-46df-abf1-307f5c7ae8ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d620a6a-a5b4-46df-abf1-307f5c7ae8ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:11:52,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:11:52,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:11:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:11:52,070 [235] [DEBUG] [app] Starting request: urn:request:1efc7f79-1869-4b79-b427-169ea9eee332 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:52,071 [231] [DEBUG] [app] Starting request: urn:request:b173920d-6a00-4765-9df5-8cfebb2e24e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:11:52,071 [235] [DEBUG] [app] Ending request: urn:request:1efc7f79-1869-4b79-b427-169ea9eee332 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1efc7f79-1869-4b79-b427-169ea9eee332', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:52,071 [231] [DEBUG] [app] Ending request: urn:request:b173920d-6a00-4765-9df5-8cfebb2e24e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b173920d-6a00-4765-9df5-8cfebb2e24e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:11:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:11:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:11:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:11:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:11:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:11:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:52,084 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:11:52,094 [231] [DEBUG] [app] Ending request: urn:request:3f987d09-7d0a-40fc-8800-386492544cea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f987d09-7d0a-40fc-8800-386492544cea', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:52,094 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:52,094 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.034 47 0.035) gunicorn-web stdout | 2023-01-17 06:11:52,098 [236] [DEBUG] [app] Ending request: urn:request:e6f0dd7e-7efe-4dda-a500-b3e07723ecd9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6f0dd7e-7efe-4dda-a500-b3e07723ecd9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:11:52,098 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:11:52,098 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:11:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.039 47 0.039) exportactionlogsworker stdout | 2023-01-17 06:11:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:11:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:28.963662+00:00 (in 33.999768 seconds) exportactionlogsworker stdout | 2023-01-17 06:11:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:54 UTC)" (scheduled at 2023-01-17 06:11:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 06:11:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 11, 54, 964346), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:11:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 54, 964346), True, datetime.datetime(2023, 1, 17, 6, 11, 54, 964346), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:11:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 11, 54, 964346), True, datetime.datetime(2023, 1, 17, 6, 11, 54, 964346), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 11, 54, 964346), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:11:54,982 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:11:54,982 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:16:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:11:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:11:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:01.636362+00:00 (in 5.989563 seconds) notificationworker stdout | 2023-01-17 06:11:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:05 UTC)" (scheduled at 2023-01-17 06:11:55.646354+00:00) notificationworker stdout | 2023-01-17 06:11:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:11:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 11, 55, 647203), True, datetime.datetime(2023, 1, 17, 6, 11, 55, 647203), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:11:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:11:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:11:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:05 UTC)" executed successfully builder stdout | 2023-01-17 06:11:58,838 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:11:58,851 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:11:58,851 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:11:58,861 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:00,938 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:12:00,988 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:12:01,632 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:12:01,635 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} notificationworker stdout | 2023-01-17 06:12:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:05.646354+00:00 (in 4.009577 seconds) notificationworker stdout | 2023-01-17 06:12:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:01 UTC)" (scheduled at 2023-01-17 06:12:01.636362+00:00) notificationworker stdout | 2023-01-17 06:12:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:12:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:12:01,638 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:12:01,640 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:12:01,642 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:12:01,998 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:12:02,367 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:12:02,456 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:12:02,472 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:12:02,912 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:12:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:12:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:19.346120+00:00 (in 14.408893 seconds) repositorygcworker stdout | 2023-01-17 06:12:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:04 UTC)" (scheduled at 2023-01-17 06:12:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:12:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:12:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 4, 937673), True, datetime.datetime(2023, 1, 17, 6, 12, 4, 937673), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:12:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:12:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:12:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:12:05,620 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:12:05,622 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:12:05,624 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:12:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:15.646354+00:00 (in 9.999578 seconds) notificationworker stdout | 2023-01-17 06:12:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:15 UTC)" (scheduled at 2023-01-17 06:12:05.646354+00:00) notificationworker stdout | 2023-01-17 06:12:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 5, 647144), True, datetime.datetime(2023, 1, 17, 6, 12, 5, 647144), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:12:06,099 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:12:06,106 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:12:06,112 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:12:06,114 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:12:06,117 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:12:06,119 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:12:06,122 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:12:06,124 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:12:06,127 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:12:06,144 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:12:06,209 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} namespacegcworker stdout | 2023-01-17 06:12:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:12:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:41.148245+00:00 (in 34.206582 seconds) namespacegcworker stdout | 2023-01-17 06:12:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:17:06 UTC)" (scheduled at 2023-01-17 06:12:06.941185+00:00) namespacegcworker stdout | 2023-01-17 06:12:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 12, 6, 942050), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:12:06,952 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 12, 6, 942050), True, datetime.datetime(2023, 1, 17, 6, 12, 6, 942050), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:12:06,956 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 12, 6, 942050), True, datetime.datetime(2023, 1, 17, 6, 12, 6, 942050), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 12, 6, 942050), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:12:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:12:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:17:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:12:07,060 [231] [DEBUG] [app] Starting request: urn:request:2291a793-cb65-4425-b1e9-e3867b5821e4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:07,060 [233] [DEBUG] [app] Starting request: urn:request:7e5c6c09-1e09-48a6-a80b-0839cae26105 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:07,065 [230] [DEBUG] [app] Starting request: urn:request:db4bd241-4267-4d12-91c7-a44e4cb0b482 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:07,065 [240] [DEBUG] [app] Starting request: urn:request:0a01dfce-2084-4ac7-b248-f34a67b54aa6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:07,065 [230] [DEBUG] [app] Ending request: urn:request:db4bd241-4267-4d12-91c7-a44e4cb0b482 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:db4bd241-4267-4d12-91c7-a44e4cb0b482', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:12:07,066 [240] [DEBUG] [app] Ending request: urn:request:0a01dfce-2084-4ac7-b248-f34a67b54aa6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a01dfce-2084-4ac7-b248-f34a67b54aa6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:12:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:12:07,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:12:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:07,070 [235] [DEBUG] [app] Starting request: urn:request:e5cf2b79-95f2-49b2-a1a4-2bd428186445 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:07,070 [231] [DEBUG] [app] Starting request: urn:request:66ff29ad-df21-404f-94c6-b4676adf5470 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:07,070 [235] [DEBUG] [app] Ending request: urn:request:e5cf2b79-95f2-49b2-a1a4-2bd428186445 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5cf2b79-95f2-49b2-a1a4-2bd428186445', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:07,070 [231] [DEBUG] [app] Ending request: urn:request:66ff29ad-df21-404f-94c6-b4676adf5470 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66ff29ad-df21-404f-94c6-b4676adf5470', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:07,087 [231] [DEBUG] [app] Ending request: urn:request:2291a793-cb65-4425-b1e9-e3867b5821e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2291a793-cb65-4425-b1e9-e3867b5821e4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:12:07,087 [233] [DEBUG] [app] Ending request: urn:request:7e5c6c09-1e09-48a6-a80b-0839cae26105 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e5c6c09-1e09-48a6-a80b-0839cae26105', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:07,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) storagereplication stdout | 2023-01-17 06:12:07,284 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:12:07,327 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:12:07,709 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:12:08,165 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:12:08,253 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:12:08,362 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:12:08,672 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 06:12:08,872 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:08,884 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:08,884 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:08,894 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:12:09,154 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:12:09,650 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:12:09,812 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:12:10,061 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:12:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:12:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:40.738568+00:00 (in 29.999500 seconds) buildlogsarchiver stdout | 2023-01-17 06:12:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:40 UTC)" (scheduled at 2023-01-17 06:12:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:12:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 12, 10, 739462), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:12:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:12:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:12:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:12:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:25.646354+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 06:12:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:25 UTC)" (scheduled at 2023-01-17 06:12:15.646354+00:00) notificationworker stdout | 2023-01-17 06:12:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 15, 647189), True, datetime.datetime(2023, 1, 17, 6, 12, 15, 647189), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:12:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:19.537472+00:00 (in 2.600029 seconds) securityworker stdout | 2023-01-17 06:12:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:46 UTC)" (scheduled at 2023-01-17 06:12:16.936988+00:00) securityworker stdout | 2023-01-17 06:12:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:46 UTC)" executed successfully builder stdout | 2023-01-17 06:12:18,905 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:18,917 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:18,917 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:18,927 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:12:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:12:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:04.936835+00:00 (in 45.590257 seconds) repositorygcworker stdout | 2023-01-17 06:12:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:19 UTC)" (scheduled at 2023-01-17 06:12:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:12:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:12:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:12:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:46.936988+00:00 (in 27.399107 seconds) securityworker stdout | 2023-01-17 06:12:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:49 UTC)" (scheduled at 2023-01-17 06:12:19.537472+00:00) securityworker stdout | 2023-01-17 06:12:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:12:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:12:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:12:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:12:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:12:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:12:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:12:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:12:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:51.348605+00:00 (in 29.999524 seconds) gcworker stdout | 2023-01-17 06:12:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:51 UTC)" (scheduled at 2023-01-17 06:12:21.348605+00:00) gcworker stdout | 2023-01-17 06:12:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726341349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:12:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:12:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:12:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:12:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:12:22,060 [233] [DEBUG] [app] Starting request: urn:request:8ec041cd-4183-4eb0-bd1f-7c4990d0f0e6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:22,060 [235] [DEBUG] [app] Starting request: urn:request:40874e2c-2d96-43fb-acf2-d668b6faca0c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:22,065 [240] [DEBUG] [app] Starting request: urn:request:99d19824-bb4b-46b0-9e9b-69084a65cfc0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:22,066 [240] [DEBUG] [app] Ending request: urn:request:99d19824-bb4b-46b0-9e9b-69084a65cfc0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99d19824-bb4b-46b0-9e9b-69084a65cfc0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:22,066 [230] [DEBUG] [app] Starting request: urn:request:50a70e69-1e58-4135-acdd-1b44331b971d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:22,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:12:22,066 [230] [DEBUG] [app] Ending request: urn:request:50a70e69-1e58-4135-acdd-1b44331b971d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50a70e69-1e58-4135-acdd-1b44331b971d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:12:22,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:22,070 [236] [DEBUG] [app] Starting request: urn:request:a47b64a4-e3f9-4cfe-a6a9-5e4459a7b052 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:22,070 [236] [DEBUG] [app] Ending request: urn:request:a47b64a4-e3f9-4cfe-a6a9-5e4459a7b052 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a47b64a4-e3f9-4cfe-a6a9-5e4459a7b052', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:22,071 [233] [DEBUG] [app] Starting request: urn:request:7975450c-1918-47b8-a33b-533231a4f59f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:22,071 [233] [DEBUG] [app] Ending request: urn:request:7975450c-1918-47b8-a33b-533231a4f59f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7975450c-1918-47b8-a33b-533231a4f59f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:22,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:22,088 [233] [DEBUG] [app] Ending request: urn:request:8ec041cd-4183-4eb0-bd1f-7c4990d0f0e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ec041cd-4183-4eb0-bd1f-7c4990d0f0e6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:22,088 [235] [DEBUG] [app] Ending request: urn:request:40874e2c-2d96-43fb-acf2-d668b6faca0c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:40874e2c-2d96-43fb-acf2-d668b6faca0c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:22,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:12:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:12:22,089 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:12:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:35.646354+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 06:12:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:35 UTC)" (scheduled at 2023-01-17 06:12:25.646354+00:00) notificationworker stdout | 2023-01-17 06:12:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 25, 647196), True, datetime.datetime(2023, 1, 17, 6, 12, 25, 647196), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:25,660 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:25,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:25,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:35 UTC)" executed successfully builder stdout | 2023-01-17 06:12:28,938 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:28,950 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:28,950 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:28,960 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:12:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:12:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:35.853396+00:00 (in 6.889245 seconds) exportactionlogsworker stdout | 2023-01-17 06:12:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:28 UTC)" (scheduled at 2023-01-17 06:12:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:12:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:12:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:28 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:12:30,974 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:12:31,023 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:12:31,640 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:12:31,643 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:12:31,646 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:12:31,648 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:12:31,651 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:12:32,034 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:12:32,403 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:12:32,491 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:12:32,504 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:12:32,948 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:12:35,627 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:12:35,629 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:12:35,631 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:12:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:45.646354+00:00 (in 9.999593 seconds) notificationworker stdout | 2023-01-17 06:12:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:45 UTC)" (scheduled at 2023-01-17 06:12:35.646354+00:00) notificationworker stdout | 2023-01-17 06:12:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 35, 647140), True, datetime.datetime(2023, 1, 17, 6, 12, 35, 647140), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:12:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:12:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:28.963662+00:00 (in 53.109747 seconds) exportactionlogsworker stdout | 2023-01-17 06:12:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:35 UTC)" (scheduled at 2023-01-17 06:12:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:12:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:12:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 35, 854364), True, datetime.datetime(2023, 1, 17, 6, 12, 35, 854364), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:12:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:12:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:12:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:12:36,111 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:12:36,114 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:12:36,122 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:12:36,124 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:12:36,126 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:12:36,129 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:12:36,131 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:12:36,134 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:12:36,136 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:12:36,175 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:12:36,237 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:12:37,060 [231] [DEBUG] [app] Starting request: urn:request:95d0f5ab-ac6d-4281-a37d-8126ba2a5f56 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:37,060 [236] [DEBUG] [app] Starting request: urn:request:fdc84f73-38b4-4f94-9d7a-03fa15d48428 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:37,065 [241] [DEBUG] [app] Starting request: urn:request:e347d110-0e44-4907-8ab0-f3a8cb491c6f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:37,066 [240] [DEBUG] [app] Starting request: urn:request:be58e09d-bf5e-4153-9ce2-9e2555a3167a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:37,066 [241] [DEBUG] [app] Ending request: urn:request:e347d110-0e44-4907-8ab0-f3a8cb491c6f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e347d110-0e44-4907-8ab0-f3a8cb491c6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:12:37,066 [240] [DEBUG] [app] Ending request: urn:request:be58e09d-bf5e-4153-9ce2-9e2555a3167a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be58e09d-bf5e-4153-9ce2-9e2555a3167a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:12:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:12:37,067 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:12:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:37,070 [231] [DEBUG] [app] Starting request: urn:request:48b21000-ed3f-4419-8ea6-785b587c2f99 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:37,071 [231] [DEBUG] [app] Ending request: urn:request:48b21000-ed3f-4419-8ea6-785b587c2f99 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48b21000-ed3f-4419-8ea6-785b587c2f99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:37,071 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:37,072 [233] [DEBUG] [app] Starting request: urn:request:99c6a5ca-39e3-4da0-82b1-53aabdfffe8d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:37,073 [233] [DEBUG] [app] Ending request: urn:request:99c6a5ca-39e3-4da0-82b1-53aabdfffe8d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:99c6a5ca-39e3-4da0-82b1-53aabdfffe8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:37,073 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:12:37,073 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:37,074 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:37,080 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:37,094 [231] [DEBUG] [app] Ending request: urn:request:95d0f5ab-ac6d-4281-a37d-8126ba2a5f56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:95d0f5ab-ac6d-4281-a37d-8126ba2a5f56', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:37,094 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.035) gunicorn-web stdout | 2023-01-17 06:12:37,095 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:12:37,095 [236] [DEBUG] [app] Ending request: urn:request:fdc84f73-38b4-4f94-9d7a-03fa15d48428 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fdc84f73-38b4-4f94-9d7a-03fa15d48428', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:37,095 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.036 47 0.036) gunicorn-web stdout | 2023-01-17 06:12:37,096 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:12:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:12:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:37.537407+00:00 (in 0.280332 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:12:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:37 UTC)" (scheduled at 2023-01-17 06:12:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:12:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:12:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 37, 257501), True, datetime.datetime(2023, 1, 17, 6, 12, 37, 257501), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:12:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:12:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:12:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:12:37,312 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:12:37,361 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} securityscanningnotificationworker stdout | 2023-01-17 06:12:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:12:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:37.256652+00:00 (in 59.718817 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:12:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:37 UTC)" (scheduled at 2023-01-17 06:12:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:12:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:12:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:12:37,744 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:12:38,201 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:12:38,288 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:12:38,386 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:12:38,708 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 06:12:38,970 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:38,983 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:38,983 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:38,994 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:12:39,191 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:12:39,667 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:12:39,847 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:12:40,073 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:12:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:12:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:10.738568+00:00 (in 29.999539 seconds) buildlogsarchiver stdout | 2023-01-17 06:12:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:10 UTC)" (scheduled at 2023-01-17 06:12:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:12:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 12, 40, 739404), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:12:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:12:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:12:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:12:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:12:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:42.941536+00:00 (in 1.792841 seconds) namespacegcworker stdout | 2023-01-17 06:12:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:41 UTC)" (scheduled at 2023-01-17 06:12:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:12:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:12:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 41, 149076), True, datetime.datetime(2023, 1, 17, 6, 12, 41, 149076), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:12:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:12:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:12:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:12:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:12:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:41.148245+00:00 (in 58.206230 seconds) namespacegcworker stdout | 2023-01-17 06:12:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:42 UTC)" (scheduled at 2023-01-17 06:12:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:12:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:12:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:13:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:12:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:55.646354+00:00 (in 9.999547 seconds) notificationworker stdout | 2023-01-17 06:12:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:55 UTC)" (scheduled at 2023-01-17 06:12:45.646354+00:00) notificationworker stdout | 2023-01-17 06:12:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 45, 647220), True, datetime.datetime(2023, 1, 17, 6, 12, 45, 647220), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:12:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:12:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:12:49.537472+00:00 (in 2.600048 seconds) securityworker stdout | 2023-01-17 06:12:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:16 UTC)" (scheduled at 2023-01-17 06:12:46.936988+00:00) securityworker stdout | 2023-01-17 06:12:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:16 UTC)" executed successfully builder stdout | 2023-01-17 06:12:48,997 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:49,009 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:49,009 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:49,019 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:12:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:16.936988+00:00 (in 27.399072 seconds) securityworker stdout | 2023-01-17 06:12:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:19 UTC)" (scheduled at 2023-01-17 06:12:49.537472+00:00) securityworker stdout | 2023-01-17 06:12:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:12:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:12:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:12:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:12:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:12:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:12:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:12:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:12:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:12:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:12:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:12:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:12:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:12:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:12:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:12:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:21.348605+00:00 (in 29.999564 seconds) gcworker stdout | 2023-01-17 06:12:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:21 UTC)" (scheduled at 2023-01-17 06:12:51.348605+00:00) gcworker stdout | 2023-01-17 06:12:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726371349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:12:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:12:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:12:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:12:52,059 [231] [DEBUG] [app] Starting request: urn:request:3e0761aa-c841-4033-8579-3ed392e166f3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:52,059 [236] [DEBUG] [app] Starting request: urn:request:40399541-93cc-4edc-a96f-2d738cba0f27 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:12:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:12:52,064 [241] [DEBUG] [app] Starting request: urn:request:d2262afd-aee1-430a-9b4a-c5941e330b9d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:52,065 [230] [DEBUG] [app] Starting request: urn:request:8d414db1-8cc1-4358-b399-dd9d41ca0cac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:12:52,065 [241] [DEBUG] [app] Ending request: urn:request:d2262afd-aee1-430a-9b4a-c5941e330b9d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2262afd-aee1-430a-9b4a-c5941e330b9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:12:52,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:12:52,065 [230] [DEBUG] [app] Ending request: urn:request:8d414db1-8cc1-4358-b399-dd9d41ca0cac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d414db1-8cc1-4358-b399-dd9d41ca0cac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:12:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:12:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:52,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:12:52,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:12:52,069 [235] [DEBUG] [app] Starting request: urn:request:aa0a4099-916b-4b16-990e-08af5b42d751 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:52,069 [231] [DEBUG] [app] Starting request: urn:request:95789ce5-658d-4e17-8e3e-62381bbfb9fe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:12:52,070 [235] [DEBUG] [app] Ending request: urn:request:aa0a4099-916b-4b16-990e-08af5b42d751 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aa0a4099-916b-4b16-990e-08af5b42d751', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:12:52,070 [231] [DEBUG] [app] Ending request: urn:request:95789ce5-658d-4e17-8e3e-62381bbfb9fe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95789ce5-658d-4e17-8e3e-62381bbfb9fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:12:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:52,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:12:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:12:52,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:12:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:12:52,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:12:52,087 [231] [DEBUG] [app] Ending request: urn:request:3e0761aa-c841-4033-8579-3ed392e166f3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3e0761aa-c841-4033-8579-3ed392e166f3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:52,087 [236] [DEBUG] [app] Ending request: urn:request:40399541-93cc-4edc-a96f-2d738cba0f27 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:40399541-93cc-4edc-a96f-2d738cba0f27', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:12:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:12:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:12:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:12:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) notificationworker stdout | 2023-01-17 06:12:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:12:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:01.636362+00:00 (in 5.989539 seconds) notificationworker stdout | 2023-01-17 06:12:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:05 UTC)" (scheduled at 2023-01-17 06:12:55.646354+00:00) notificationworker stdout | 2023-01-17 06:12:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:12:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 12, 55, 647241), True, datetime.datetime(2023, 1, 17, 6, 12, 55, 647241), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:12:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:12:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:12:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:05 UTC)" executed successfully builder stdout | 2023-01-17 06:12:59,029 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:12:59,049 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:12:59,049 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:12:59,060 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:01,009 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:13:01,035 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:13:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:05.646354+00:00 (in 4.009533 seconds) notificationworker stdout | 2023-01-17 06:13:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:01 UTC)" (scheduled at 2023-01-17 06:13:01.636362+00:00) notificationworker stdout | 2023-01-17 06:13:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:13:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:13:01,648 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:13:01,650 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:13:01,652 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:13:01,655 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:13:01,658 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:13:02,049 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:13:02,437 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:13:02,516 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:13:02,520 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:13:02,961 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:13:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:13:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:19.346120+00:00 (in 14.408842 seconds) repositorygcworker stdout | 2023-01-17 06:13:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:04 UTC)" (scheduled at 2023-01-17 06:13:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:13:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:13:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 4, 937740), True, datetime.datetime(2023, 1, 17, 6, 13, 4, 937740), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:13:04,950 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:13:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:13:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:04 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:13:05,634 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:13:05,636 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:13:05,639 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:13:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:15.646354+00:00 (in 9.999533 seconds) notificationworker stdout | 2023-01-17 06:13:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:15 UTC)" (scheduled at 2023-01-17 06:13:05.646354+00:00) notificationworker stdout | 2023-01-17 06:13:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 5, 647209), True, datetime.datetime(2023, 1, 17, 6, 13, 5, 647209), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:13:06,122 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:13:06,125 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:13:06,130 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:13:06,133 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:13:06,135 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:13:06,138 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:13:06,141 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:13:06,143 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:13:06,146 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:13:06,187 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:13:06,272 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:13:07,060 [235] [DEBUG] [app] Starting request: urn:request:d1517c03-a380-4475-b643-d66d8658003e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:07,060 [231] [DEBUG] [app] Starting request: urn:request:21943a14-e3ba-441e-9c96-e6e1ceb0bd93 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:07,064 [243] [DEBUG] [app] Starting request: urn:request:65b7b623-be7a-4977-baa8-908a32ab5501 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:07,065 [243] [DEBUG] [app] Ending request: urn:request:65b7b623-be7a-4977-baa8-908a32ab5501 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:65b7b623-be7a-4977-baa8-908a32ab5501', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:07,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:07,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:13:07,065 [230] [DEBUG] [app] Starting request: urn:request:6cee848f-1b14-4415-9ed1-822d3a0718ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:07,066 [230] [DEBUG] [app] Ending request: urn:request:6cee848f-1b14-4415-9ed1-822d3a0718ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6cee848f-1b14-4415-9ed1-822d3a0718ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:13:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:07,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:07,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:07,069 [233] [DEBUG] [app] Starting request: urn:request:5f9377c6-5493-4992-8c9f-2616ed850c6a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:07,070 [233] [DEBUG] [app] Ending request: urn:request:5f9377c6-5493-4992-8c9f-2616ed850c6a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f9377c6-5493-4992-8c9f-2616ed850c6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:07,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:07,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:07,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:07,071 [236] [DEBUG] [app] Starting request: urn:request:43f569a3-6201-4362-a4d8-c687915d811b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:07,072 [236] [DEBUG] [app] Ending request: urn:request:43f569a3-6201-4362-a4d8-c687915d811b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:43f569a3-6201-4362-a4d8-c687915d811b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:07,073 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:07,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:07,087 [235] [DEBUG] [app] Ending request: urn:request:d1517c03-a380-4475-b643-d66d8658003e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1517c03-a380-4475-b643-d66d8658003e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:13:07,088 [231] [DEBUG] [app] Ending request: urn:request:21943a14-e3ba-441e-9c96-e6e1ceb0bd93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21943a14-e3ba-441e-9c96-e6e1ceb0bd93', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) storagereplication stdout | 2023-01-17 06:13:07,347 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:13:07,395 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:13:07,750 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:13:08,237 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:13:08,323 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:13:08,412 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:13:08,728 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 06:13:09,060 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:09,073 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:09,073 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:09,083 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:13:09,208 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:13:09,687 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:13:09,854 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:13:10,090 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:13:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:13:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:40.738568+00:00 (in 29.999562 seconds) buildlogsarchiver stdout | 2023-01-17 06:13:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:40 UTC)" (scheduled at 2023-01-17 06:13:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:13:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 13, 10, 739446), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:13:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:13:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:13:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:13:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:25.646354+00:00 (in 9.999596 seconds) notificationworker stdout | 2023-01-17 06:13:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:25 UTC)" (scheduled at 2023-01-17 06:13:15.646354+00:00) notificationworker stdout | 2023-01-17 06:13:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 15, 647214), True, datetime.datetime(2023, 1, 17, 6, 13, 15, 647214), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:13:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:19.537472+00:00 (in 2.600003 seconds) securityworker stdout | 2023-01-17 06:13:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:46 UTC)" (scheduled at 2023-01-17 06:13:16.936988+00:00) securityworker stdout | 2023-01-17 06:13:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:46 UTC)" executed successfully builder stdout | 2023-01-17 06:13:19,094 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:19,106 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:19,106 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:19,117 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:13:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:13:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:04.936835+00:00 (in 45.590277 seconds) repositorygcworker stdout | 2023-01-17 06:13:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:19 UTC)" (scheduled at 2023-01-17 06:13:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:13:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:13:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:13:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:46.936988+00:00 (in 27.399023 seconds) securityworker stdout | 2023-01-17 06:13:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:49 UTC)" (scheduled at 2023-01-17 06:13:19.537472+00:00) securityworker stdout | 2023-01-17 06:13:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:13:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:13:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:13:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:13:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:13:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:13:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:13:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:13:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:51.348605+00:00 (in 29.999533 seconds) gcworker stdout | 2023-01-17 06:13:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:51 UTC)" (scheduled at 2023-01-17 06:13:21.348605+00:00) gcworker stdout | 2023-01-17 06:13:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726401349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:13:21,370 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:13:21,370 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:13:21,370 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:13:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:13:22,059 [233] [DEBUG] [app] Starting request: urn:request:96eef874-af62-4113-aaf6-25d74f20b7ef (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:22,060 [231] [DEBUG] [app] Starting request: urn:request:bebc2979-a014-43f3-adcc-79b5bcd43c37 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:22,064 [230] [DEBUG] [app] Starting request: urn:request:78e7d48e-0a73-4946-a305-e5e05a5a5ddb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:22,065 [242] [DEBUG] [app] Starting request: urn:request:7622a300-cab6-4e0b-9186-a47a3f226a4a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:22,065 [230] [DEBUG] [app] Ending request: urn:request:78e7d48e-0a73-4946-a305-e5e05a5a5ddb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78e7d48e-0a73-4946-a305-e5e05a5a5ddb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:13:22,065 [242] [DEBUG] [app] Ending request: urn:request:7622a300-cab6-4e0b-9186-a47a3f226a4a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7622a300-cab6-4e0b-9186-a47a3f226a4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:22,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:13:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:13:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:22,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:22,069 [236] [DEBUG] [app] Starting request: urn:request:45b464c5-2302-43ae-b2e2-4ccd1559918b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:22,070 [235] [DEBUG] [app] Starting request: urn:request:035eeb37-379d-4e8e-91ec-69dffdb56175 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:22,070 [236] [DEBUG] [app] Ending request: urn:request:45b464c5-2302-43ae-b2e2-4ccd1559918b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45b464c5-2302-43ae-b2e2-4ccd1559918b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:22,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:22,070 [235] [DEBUG] [app] Ending request: urn:request:035eeb37-379d-4e8e-91ec-69dffdb56175 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:035eeb37-379d-4e8e-91ec-69dffdb56175', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:22,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:22,087 [231] [DEBUG] [app] Ending request: urn:request:bebc2979-a014-43f3-adcc-79b5bcd43c37 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bebc2979-a014-43f3-adcc-79b5bcd43c37', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:22,087 [233] [DEBUG] [app] Ending request: urn:request:96eef874-af62-4113-aaf6-25d74f20b7ef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:96eef874-af62-4113-aaf6-25d74f20b7ef', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:22,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:22,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:13:22,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) notificationworker stdout | 2023-01-17 06:13:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:35.646354+00:00 (in 9.999568 seconds) notificationworker stdout | 2023-01-17 06:13:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:35 UTC)" (scheduled at 2023-01-17 06:13:25.646354+00:00) notificationworker stdout | 2023-01-17 06:13:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 25, 647213), True, datetime.datetime(2023, 1, 17, 6, 13, 25, 647213), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:13:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:13:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:35.853396+00:00 (in 6.889226 seconds) exportactionlogsworker stdout | 2023-01-17 06:13:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:28 UTC)" (scheduled at 2023-01-17 06:13:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:13:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:13:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:28 UTC)" executed successfully builder stdout | 2023-01-17 06:13:29,118 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:29,130 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:29,130 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:29,141 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:31,045 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gcworker stdout | 2023-01-17 06:13:31,048 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} gunicorn-web stdout | 2023-01-17 06:13:31,655 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:13:31,657 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:13:31,660 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:13:31,663 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:13:31,666 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:13:32,058 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:13:32,444 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:13:32,536 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:13:32,551 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:13:32,985 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-secscan stdout | 2023-01-17 06:13:35,642 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:13:35,645 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} notificationworker stdout | 2023-01-17 06:13:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:36.635997+00:00 (in 0.989217 seconds) notificationworker stdout | 2023-01-17 06:13:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:45 UTC)" (scheduled at 2023-01-17 06:13:35.646354+00:00) notificationworker stdout | 2023-01-17 06:13:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. gunicorn-secscan stdout | 2023-01-17 06:13:35,647 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:13:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 35, 647176), True, datetime.datetime(2023, 1, 17, 6, 13, 35, 647176), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:45 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:13:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:13:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:28.963662+00:00 (in 53.109721 seconds) exportactionlogsworker stdout | 2023-01-17 06:13:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:35 UTC)" (scheduled at 2023-01-17 06:13:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:13:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:13:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 35, 854423), True, datetime.datetime(2023, 1, 17, 6, 13, 35, 854423), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:13:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:13:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:13:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:13:36,133 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:13:36,137 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:13:36,139 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:13:36,141 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:13:36,144 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:13:36,146 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:13:36,149 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:13:36,151 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:13:36,155 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:13:36,197 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:13:36,280 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 06:13:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:45.646354+00:00 (in 9.009918 seconds) notificationworker stdout | 2023-01-17 06:13:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:18:36 UTC)" (scheduled at 2023-01-17 06:13:36.635997+00:00) notificationworker stdout | 2023-01-17 06:13:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 13, 36, 636839), 'notification/%']) notificationworker stdout | 2023-01-17 06:13:36,647 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 13, 36, 636839), True, datetime.datetime(2023, 1, 17, 6, 13, 36, 636839), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:13:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 13, 36, 636839), True, datetime.datetime(2023, 1, 17, 6, 13, 36, 636839), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 13, 36, 636839), 'notification/%']) notificationworker stdout | 2023-01-17 06:13:36,654 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:36,654 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:18:36 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:13:37,060 [236] [DEBUG] [app] Starting request: urn:request:d7a14d67-b7d0-41d7-9362-6b4d89121ee8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:37,060 [231] [DEBUG] [app] Starting request: urn:request:ad404fae-96a2-4f99-b04e-6aa0a8f113db (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:37,065 [230] [DEBUG] [app] Starting request: urn:request:0327080e-ac2f-4336-8f9c-78ed08313c79 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:37,066 [243] [DEBUG] [app] Starting request: urn:request:bece0e71-c2c9-4320-9335-b7e431c6e202 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:37,066 [230] [DEBUG] [app] Ending request: urn:request:0327080e-ac2f-4336-8f9c-78ed08313c79 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0327080e-ac2f-4336-8f9c-78ed08313c79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:13:37,066 [243] [DEBUG] [app] Ending request: urn:request:bece0e71-c2c9-4320-9335-b7e431c6e202 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bece0e71-c2c9-4320-9335-b7e431c6e202', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:13:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:13:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:13:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:37,070 [235] [DEBUG] [app] Starting request: urn:request:e04ff80d-66e2-499b-947a-abe348579c81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:37,071 [235] [DEBUG] [app] Ending request: urn:request:e04ff80d-66e2-499b-947a-abe348579c81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e04ff80d-66e2-499b-947a-abe348579c81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:37,071 [233] [DEBUG] [app] Starting request: urn:request:5a59c6c9-8f4e-4a4c-a358-9c0afdeef786 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:37,071 [233] [DEBUG] [app] Ending request: urn:request:5a59c6c9-8f4e-4a4c-a358-9c0afdeef786 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a59c6c9-8f4e-4a4c-a358-9c0afdeef786', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:37,082 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:37,088 [236] [DEBUG] [app] Ending request: urn:request:d7a14d67-b7d0-41d7-9362-6b4d89121ee8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d7a14d67-b7d0-41d7-9362-6b4d89121ee8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:13:37,091 [231] [DEBUG] [app] Ending request: urn:request:ad404fae-96a2-4f99-b04e-6aa0a8f113db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ad404fae-96a2-4f99-b04e-6aa0a8f113db', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:37,091 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:37,091 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) securityscanningnotificationworker stdout | 2023-01-17 06:13:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:13:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:37.537407+00:00 (in 0.280332 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:13:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:37 UTC)" (scheduled at 2023-01-17 06:13:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:13:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:13:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 37, 257475), True, datetime.datetime(2023, 1, 17, 6, 13, 37, 257475), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:13:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:13:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:13:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:13:37,383 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:13:37,430 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} securityscanningnotificationworker stdout | 2023-01-17 06:13:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:13:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:37.256652+00:00 (in 59.718834 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:13:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:37 UTC)" (scheduled at 2023-01-17 06:13:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:13:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:13:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:13:37,780 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:13:38,253 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:13:38,350 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:13:38,447 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:13:38,759 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} builder stdout | 2023-01-17 06:13:39,151 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:39,163 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:39,163 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:39,174 [60] [DEBUG] [data.database] Disconnecting from database. globalpromstats stdout | 2023-01-17 06:13:39,232 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:13:39,718 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:13:39,878 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:13:40,114 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:13:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:13:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:10.738568+00:00 (in 29.999498 seconds) buildlogsarchiver stdout | 2023-01-17 06:13:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:10 UTC)" (scheduled at 2023-01-17 06:13:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:13:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 13, 40, 739529), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:13:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:13:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:13:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:13:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:13:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:42.941536+00:00 (in 1.792754 seconds) namespacegcworker stdout | 2023-01-17 06:13:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:41 UTC)" (scheduled at 2023-01-17 06:13:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:13:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:13:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 41, 149247), True, datetime.datetime(2023, 1, 17, 6, 13, 41, 149247), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:13:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:13:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:13:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:13:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:13:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:41.148245+00:00 (in 58.206197 seconds) namespacegcworker stdout | 2023-01-17 06:13:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:42 UTC)" (scheduled at 2023-01-17 06:13:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:13:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:13:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:14:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:13:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:55.646354+00:00 (in 9.999516 seconds) notificationworker stdout | 2023-01-17 06:13:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:55 UTC)" (scheduled at 2023-01-17 06:13:45.646354+00:00) notificationworker stdout | 2023-01-17 06:13:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 45, 647301), True, datetime.datetime(2023, 1, 17, 6, 13, 45, 647301), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:13:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:13:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:13:49.537472+00:00 (in 2.600008 seconds) securityworker stdout | 2023-01-17 06:13:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:16 UTC)" (scheduled at 2023-01-17 06:13:46.936988+00:00) securityworker stdout | 2023-01-17 06:13:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:16 UTC)" executed successfully builder stdout | 2023-01-17 06:13:49,184 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:49,197 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:49,197 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:49,207 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:13:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:16.936988+00:00 (in 27.399064 seconds) securityworker stdout | 2023-01-17 06:13:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:19 UTC)" (scheduled at 2023-01-17 06:13:49.537472+00:00) securityworker stdout | 2023-01-17 06:13:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:13:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:13:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:13:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:13:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:13:49,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:13:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:13:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:13:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:13:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:13:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:13:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:13:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:13:49,555 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:13:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:13:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:21.348605+00:00 (in 29.999512 seconds) gcworker stdout | 2023-01-17 06:13:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:21 UTC)" (scheduled at 2023-01-17 06:13:51.348605+00:00) gcworker stdout | 2023-01-17 06:13:51,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 06:13:51,360 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726431360, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:13:51,365 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:13:51,365 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:13:51,365 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:13:52,059 [236] [DEBUG] [app] Starting request: urn:request:c8d6ae7a-d84d-4007-b9e8-0dd454db3e1d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:52,060 [231] [DEBUG] [app] Starting request: urn:request:20654311-4208-41b8-81e9-544a0124d829 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:13:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:13:52,064 [242] [DEBUG] [app] Starting request: urn:request:0d002f04-eb34-4baf-bac8-90a24f8b1b12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:52,065 [230] [DEBUG] [app] Starting request: urn:request:dcb8c651-2e5e-441c-9fee-9e2e09ad9932 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:13:52,065 [242] [DEBUG] [app] Ending request: urn:request:0d002f04-eb34-4baf-bac8-90a24f8b1b12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d002f04-eb34-4baf-bac8-90a24f8b1b12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:52,065 [230] [DEBUG] [app] Ending request: urn:request:dcb8c651-2e5e-441c-9fee-9e2e09ad9932 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dcb8c651-2e5e-441c-9fee-9e2e09ad9932', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:13:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:13:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:13:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:13:52,070 [235] [DEBUG] [app] Starting request: urn:request:03f11317-2024-4fdf-902e-7b795051e4cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:52,070 [233] [DEBUG] [app] Starting request: urn:request:9a115098-8e5c-49f9-8ee3-c34dd3965b1a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:13:52,070 [235] [DEBUG] [app] Ending request: urn:request:03f11317-2024-4fdf-902e-7b795051e4cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03f11317-2024-4fdf-902e-7b795051e4cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:52,071 [233] [DEBUG] [app] Ending request: urn:request:9a115098-8e5c-49f9-8ee3-c34dd3965b1a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9a115098-8e5c-49f9-8ee3-c34dd3965b1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:13:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:13:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:13:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:13:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:13:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:13:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:13:52,087 [231] [DEBUG] [app] Ending request: urn:request:20654311-4208-41b8-81e9-544a0124d829 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20654311-4208-41b8-81e9-544a0124d829', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:52,087 [236] [DEBUG] [app] Ending request: urn:request:c8d6ae7a-d84d-4007-b9e8-0dd454db3e1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c8d6ae7a-d84d-4007-b9e8-0dd454db3e1d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:13:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:13:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:13:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:13:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:13:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:13:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:01.636362+00:00 (in 5.989557 seconds) notificationworker stdout | 2023-01-17 06:13:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:05 UTC)" (scheduled at 2023-01-17 06:13:55.646354+00:00) notificationworker stdout | 2023-01-17 06:13:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:13:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 13, 55, 647258), True, datetime.datetime(2023, 1, 17, 6, 13, 55, 647258), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:13:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:13:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:13:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:05 UTC)" executed successfully builder stdout | 2023-01-17 06:13:59,217 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:13:59,230 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:13:59,230 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:13:59,240 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:14:01,078 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:14:01,080 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:14:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:05.646354+00:00 (in 4.009510 seconds) notificationworker stdout | 2023-01-17 06:14:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:01 UTC)" (scheduled at 2023-01-17 06:14:01.636362+00:00) notificationworker stdout | 2023-01-17 06:14:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:14:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:14:01,664 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:14:01,666 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:14:01,669 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:14:01,672 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:14:01,674 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:14:02,087 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:14:02,454 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:14:02,571 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:14:02,585 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:14:03,021 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:14:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:14:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:19.346120+00:00 (in 14.408890 seconds) repositorygcworker stdout | 2023-01-17 06:14:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:04 UTC)" (scheduled at 2023-01-17 06:14:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:14:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:14:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 4, 937622), True, datetime.datetime(2023, 1, 17, 6, 14, 4, 937622), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:14:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:14:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:14:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:14:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:15.646354+00:00 (in 9.999534 seconds) notificationworker stdout | 2023-01-17 06:14:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:15 UTC)" (scheduled at 2023-01-17 06:14:05.646354+00:00) notificationworker stdout | 2023-01-17 06:14:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 5, 647243), True, datetime.datetime(2023, 1, 17, 6, 14, 5, 647243), 0, 'notification/%', 50, 1, 0]) gunicorn-secscan stdout | 2023-01-17 06:14:05,650 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:14:05,653 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:14:05,655 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} notificationworker stdout | 2023-01-17 06:14:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:15 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:14:06,143 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:14:06,145 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:14:06,149 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:14:06,152 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:14:06,154 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:14:06,157 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:14:06,159 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:14:06,161 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:14:06,165 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:14:06,232 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:14:06,299 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:14:07,060 [231] [DEBUG] [app] Starting request: urn:request:0cdd32e9-1740-43bd-b2b5-48e8333d2ffe (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:07,060 [236] [DEBUG] [app] Starting request: urn:request:8286ffc4-00db-4a78-8caf-dccdbf4bbfde (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:07,065 [242] [DEBUG] [app] Starting request: urn:request:68b6681d-ff0b-45b0-bb0c-d4873a0554bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:07,065 [243] [DEBUG] [app] Starting request: urn:request:016b92e1-b911-463b-a34b-50782446adba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:07,065 [242] [DEBUG] [app] Ending request: urn:request:68b6681d-ff0b-45b0-bb0c-d4873a0554bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68b6681d-ff0b-45b0-bb0c-d4873a0554bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:14:07,065 [243] [DEBUG] [app] Ending request: urn:request:016b92e1-b911-463b-a34b-50782446adba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:016b92e1-b911-463b-a34b-50782446adba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:14:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:14:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:14:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:07,069 [236] [DEBUG] [app] Starting request: urn:request:8aed94fb-1a75-457e-8026-c22eeec9e0cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:07,070 [236] [DEBUG] [app] Ending request: urn:request:8aed94fb-1a75-457e-8026-c22eeec9e0cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8aed94fb-1a75-457e-8026-c22eeec9e0cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:07,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:07,071 [235] [DEBUG] [app] Starting request: urn:request:6a7f7e15-1c4f-437d-be3b-e424f8615a4a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:07,072 [235] [DEBUG] [app] Ending request: urn:request:6a7f7e15-1c4f-437d-be3b-e424f8615a4a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a7f7e15-1c4f-437d-be3b-e424f8615a4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:14:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:07,073 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:07,086 [231] [DEBUG] [app] Ending request: urn:request:0cdd32e9-1740-43bd-b2b5-48e8333d2ffe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0cdd32e9-1740-43bd-b2b5-48e8333d2ffe', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:07,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:14:07,087 [236] [DEBUG] [app] Ending request: urn:request:8286ffc4-00db-4a78-8caf-dccdbf4bbfde (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8286ffc4-00db-4a78-8caf-dccdbf4bbfde', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) storagereplication stdout | 2023-01-17 06:14:07,419 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:14:07,445 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:14:07,816 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:14:08,287 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:14:08,356 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:14:08,483 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:14:08,791 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:14:09,240 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:14:09,247 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:09,259 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:09,259 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:09,270 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:14:09,754 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:14:09,886 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:14:10,149 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:14:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:14:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:40.738568+00:00 (in 29.999429 seconds) buildlogsarchiver stdout | 2023-01-17 06:14:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:40 UTC)" (scheduled at 2023-01-17 06:14:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:14:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 14, 10, 739617), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:14:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:14:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:14:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:14:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:25.646354+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 06:14:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:25 UTC)" (scheduled at 2023-01-17 06:14:15.646354+00:00) notificationworker stdout | 2023-01-17 06:14:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 15, 647263), True, datetime.datetime(2023, 1, 17, 6, 14, 15, 647263), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:14:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:19.537472+00:00 (in 2.600030 seconds) securityworker stdout | 2023-01-17 06:14:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:46 UTC)" (scheduled at 2023-01-17 06:14:16.936988+00:00) securityworker stdout | 2023-01-17 06:14:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:46 UTC)" executed successfully builder stdout | 2023-01-17 06:14:19,280 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:19,292 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:19,292 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:19,302 [60] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:14:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:14:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:04.936835+00:00 (in 45.590267 seconds) repositorygcworker stdout | 2023-01-17 06:14:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:19 UTC)" (scheduled at 2023-01-17 06:14:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:14:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:14:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:14:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:46.936988+00:00 (in 27.399055 seconds) securityworker stdout | 2023-01-17 06:14:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:49 UTC)" (scheduled at 2023-01-17 06:14:19.537472+00:00) securityworker stdout | 2023-01-17 06:14:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:14:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:14:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:14:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:14:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:14:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:14:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:14:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:14:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:51.348605+00:00 (in 29.999500 seconds) gcworker stdout | 2023-01-17 06:14:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:51 UTC)" (scheduled at 2023-01-17 06:14:21.348605+00:00) gcworker stdout | 2023-01-17 06:14:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726461349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:14:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:14:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:14:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:14:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:14:22,060 [231] [DEBUG] [app] Starting request: urn:request:486eaf45-d4cf-46ea-bc2f-9d700cdb2494 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:22,060 [235] [DEBUG] [app] Starting request: urn:request:c25c5642-5060-4029-86fd-79f24a1486c2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:22,065 [242] [DEBUG] [app] Starting request: urn:request:f4a962f0-a776-46dd-811a-8a262a264df5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:22,065 [230] [DEBUG] [app] Starting request: urn:request:31e10c00-9582-40c4-83c4-005112077755 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:22,065 [242] [DEBUG] [app] Ending request: urn:request:f4a962f0-a776-46dd-811a-8a262a264df5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4a962f0-a776-46dd-811a-8a262a264df5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:14:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:14:22,066 [230] [DEBUG] [app] Ending request: urn:request:31e10c00-9582-40c4-83c4-005112077755 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31e10c00-9582-40c4-83c4-005112077755', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:14:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:14:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:22,070 [236] [DEBUG] [app] Starting request: urn:request:34bb580f-108c-432f-aa27-9cd4b9591472 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:22,070 [233] [DEBUG] [app] Starting request: urn:request:335b98b9-63f7-4524-b456-290ad9cc9ca0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:22,070 [236] [DEBUG] [app] Ending request: urn:request:34bb580f-108c-432f-aa27-9cd4b9591472 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:34bb580f-108c-432f-aa27-9cd4b9591472', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:22,071 [233] [DEBUG] [app] Ending request: urn:request:335b98b9-63f7-4524-b456-290ad9cc9ca0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:335b98b9-63f7-4524-b456-290ad9cc9ca0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:22,088 [235] [DEBUG] [app] Ending request: urn:request:c25c5642-5060-4029-86fd-79f24a1486c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c25c5642-5060-4029-86fd-79f24a1486c2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:22,088 [231] [DEBUG] [app] Ending request: urn:request:486eaf45-d4cf-46ea-bc2f-9d700cdb2494 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:486eaf45-d4cf-46ea-bc2f-9d700cdb2494', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:14:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 06:14:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:35.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 06:14:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:35 UTC)" (scheduled at 2023-01-17 06:14:25.646354+00:00) notificationworker stdout | 2023-01-17 06:14:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 25, 647263), True, datetime.datetime(2023, 1, 17, 6, 14, 25, 647263), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:25,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:14:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:14:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:35.853396+00:00 (in 6.889175 seconds) exportactionlogsworker stdout | 2023-01-17 06:14:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:28 UTC)" (scheduled at 2023-01-17 06:14:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:14:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:14:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:28 UTC)" executed successfully builder stdout | 2023-01-17 06:14:29,312 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:29,324 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:29,324 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:29,335 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:14:31,100 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:14:31,115 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:14:31,670 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:14:31,674 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:14:31,676 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:14:31,678 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:14:31,681 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:14:32,094 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:14:32,474 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:14:32,589 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:14:32,600 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:14:33,049 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:14:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:45.646354+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 06:14:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:45 UTC)" (scheduled at 2023-01-17 06:14:35.646354+00:00) notificationworker stdout | 2023-01-17 06:14:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 35, 647253), True, datetime.datetime(2023, 1, 17, 6, 14, 35, 647253), 0, 'notification/%', 50, 1, 0]) gunicorn-secscan stdout | 2023-01-17 06:14:35,657 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} notificationworker stdout | 2023-01-17 06:14:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:14:35,660 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:14:35,662 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:14:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:14:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:28.963662+00:00 (in 53.109750 seconds) exportactionlogsworker stdout | 2023-01-17 06:14:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:35 UTC)" (scheduled at 2023-01-17 06:14:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:14:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:14:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 35, 854265), True, datetime.datetime(2023, 1, 17, 6, 14, 35, 854265), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:14:35,866 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:14:35,866 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:14:35,867 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:14:36,149 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:14:36,152 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:14:36,155 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:14:36,161 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:14:36,163 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:14:36,167 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:14:36,169 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:14:36,172 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:14:36,174 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:14:36,246 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:14:36,334 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:14:37,060 [235] [DEBUG] [app] Starting request: urn:request:d5cb37d1-8397-4060-afbc-ecc18988afeb (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:37,060 [231] [DEBUG] [app] Starting request: urn:request:2ca0bab9-b501-4f5f-b04b-ecb49a632561 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:37,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:37,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:37,065 [242] [DEBUG] [app] Starting request: urn:request:4b17ee34-21a2-43a2-b6b5-c8d653ce6db6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:37,066 [230] [DEBUG] [app] Starting request: urn:request:2dcce60d-1d34-4e70-b934-ff65efd42d91 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:37,066 [242] [DEBUG] [app] Ending request: urn:request:4b17ee34-21a2-43a2-b6b5-c8d653ce6db6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4b17ee34-21a2-43a2-b6b5-c8d653ce6db6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:14:37,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:14:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:14:37,066 [230] [DEBUG] [app] Ending request: urn:request:2dcce60d-1d34-4e70-b934-ff65efd42d91 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2dcce60d-1d34-4e70-b934-ff65efd42d91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:14:37,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:37,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:37,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:37,071 [236] [DEBUG] [app] Starting request: urn:request:adcb3caf-2365-4439-bae5-fa661b7708c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:37,071 [236] [DEBUG] [app] Ending request: urn:request:adcb3caf-2365-4439-bae5-fa661b7708c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:adcb3caf-2365-4439-bae5-fa661b7708c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:37,071 [233] [DEBUG] [app] Starting request: urn:request:c6335982-5862-48e2-9c7d-2a547df0a1de (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:37,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:37,072 [233] [DEBUG] [app] Ending request: urn:request:c6335982-5862-48e2-9c7d-2a547df0a1de (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6335982-5862-48e2-9c7d-2a547df0a1de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:37,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:14:37,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:37,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:37,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:37,088 [235] [DEBUG] [app] Ending request: urn:request:d5cb37d1-8397-4060-afbc-ecc18988afeb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d5cb37d1-8397-4060-afbc-ecc18988afeb', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:14:37,088 [231] [DEBUG] [app] Ending request: urn:request:2ca0bab9-b501-4f5f-b04b-ecb49a632561 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2ca0bab9-b501-4f5f-b04b-ecb49a632561', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:37,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:14:37,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:14:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:14:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:37.537407+00:00 (in 0.280339 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:14:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:37 UTC)" (scheduled at 2023-01-17 06:14:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:14:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:14:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 37, 257478), True, datetime.datetime(2023, 1, 17, 6, 14, 37, 257478), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:14:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:14:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:14:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:14:37,454 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:14:37,461 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} securityscanningnotificationworker stdout | 2023-01-17 06:14:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:14:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:37.256652+00:00 (in 59.718839 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:14:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:37 UTC)" (scheduled at 2023-01-17 06:14:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:14:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:14:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:14:37,850 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:14:38,300 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:14:38,372 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:14:38,508 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:14:38,824 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:14:39,264 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:14:39,345 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:39,357 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:39,357 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:39,367 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:14:39,784 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:14:39,912 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:14:40,181 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:14:40,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:14:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:10.738568+00:00 (in 29.999423 seconds) buildlogsarchiver stdout | 2023-01-17 06:14:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:10 UTC)" (scheduled at 2023-01-17 06:14:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:14:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 14, 40, 739555), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:14:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:14:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:14:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:14:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:14:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:42.941536+00:00 (in 1.792817 seconds) namespacegcworker stdout | 2023-01-17 06:14:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:41 UTC)" (scheduled at 2023-01-17 06:14:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:14:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:14:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 41, 149113), True, datetime.datetime(2023, 1, 17, 6, 14, 41, 149113), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:14:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:14:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:14:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:14:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:14:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:41.148245+00:00 (in 58.206207 seconds) namespacegcworker stdout | 2023-01-17 06:14:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:42 UTC)" (scheduled at 2023-01-17 06:14:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:14:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:14:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:15:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:14:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:55.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 06:14:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:55 UTC)" (scheduled at 2023-01-17 06:14:45.646354+00:00) notificationworker stdout | 2023-01-17 06:14:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 45, 647179), True, datetime.datetime(2023, 1, 17, 6, 14, 45, 647179), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:45,661 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:45,661 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:45,661 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:14:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:14:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:14:49.537472+00:00 (in 2.600023 seconds) securityworker stdout | 2023-01-17 06:14:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:16 UTC)" (scheduled at 2023-01-17 06:14:46.936988+00:00) securityworker stdout | 2023-01-17 06:14:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:16 UTC)" executed successfully builder stdout | 2023-01-17 06:14:49,377 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:49,389 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:49,389 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:49,400 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:14:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:16.936988+00:00 (in 27.399046 seconds) securityworker stdout | 2023-01-17 06:14:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:19 UTC)" (scheduled at 2023-01-17 06:14:49.537472+00:00) securityworker stdout | 2023-01-17 06:14:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:14:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:14:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:14:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:14:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:14:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:14:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:14:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:14:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:14:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:14:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:14:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:14:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:14:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:19 UTC)" executed successfully gcworker stdout | 2023-01-17 06:14:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:14:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:21.348605+00:00 (in 29.999475 seconds) gcworker stdout | 2023-01-17 06:14:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:21 UTC)" (scheduled at 2023-01-17 06:14:51.348605+00:00) gcworker stdout | 2023-01-17 06:14:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726491349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:14:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:14:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:14:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:14:52,060 [233] [DEBUG] [app] Starting request: urn:request:6e5ca5f3-cf40-4337-9652-80fe645afb61 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:52,060 [236] [DEBUG] [app] Starting request: urn:request:a85e0344-da71-4a81-9a11-6cf9bb308f34 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:14:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:52,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:14:52,065 [243] [DEBUG] [app] Starting request: urn:request:2827f0a9-9eb8-4882-9add-b8ee83d40643 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:52,066 [242] [DEBUG] [app] Starting request: urn:request:23689831-5446-495b-b298-b5a13fdfbdde (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:14:52,066 [243] [DEBUG] [app] Ending request: urn:request:2827f0a9-9eb8-4882-9add-b8ee83d40643 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2827f0a9-9eb8-4882-9add-b8ee83d40643', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:14:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:14:52,066 [242] [DEBUG] [app] Ending request: urn:request:23689831-5446-495b-b298-b5a13fdfbdde (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23689831-5446-495b-b298-b5a13fdfbdde', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:14:52,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:52,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:14:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:14:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:52,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:14:52,070 [231] [DEBUG] [app] Starting request: urn:request:178d9e67-84e9-4155-8d3c-febde8120773 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:52,071 [236] [DEBUG] [app] Starting request: urn:request:45247004-b0e5-4706-8eab-621d3050f217 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:14:52,071 [231] [DEBUG] [app] Ending request: urn:request:178d9e67-84e9-4155-8d3c-febde8120773 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:178d9e67-84e9-4155-8d3c-febde8120773', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:52,071 [236] [DEBUG] [app] Ending request: urn:request:45247004-b0e5-4706-8eab-621d3050f217 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45247004-b0e5-4706-8eab-621d3050f217', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:14:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:14:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:14:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:14:52,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:14:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:52,079 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:14:52,088 [236] [DEBUG] [app] Ending request: urn:request:a85e0344-da71-4a81-9a11-6cf9bb308f34 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a85e0344-da71-4a81-9a11-6cf9bb308f34', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:14:52,088 [233] [DEBUG] [app] Ending request: urn:request:6e5ca5f3-cf40-4337-9652-80fe645afb61 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e5ca5f3-cf40-4337-9652-80fe645afb61', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:14:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:14:52,088 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:14:52,089 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:14:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:14:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:14:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:01.636362+00:00 (in 5.989567 seconds) notificationworker stdout | 2023-01-17 06:14:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:05 UTC)" (scheduled at 2023-01-17 06:14:55.646354+00:00) notificationworker stdout | 2023-01-17 06:14:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:14:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 14, 55, 647184), True, datetime.datetime(2023, 1, 17, 6, 14, 55, 647184), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:14:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:14:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:14:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:05 UTC)" executed successfully builder stdout | 2023-01-17 06:14:59,410 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:14:59,422 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:14:59,422 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:14:59,432 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:15:01,131 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:15:01,150 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:15:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:05.646354+00:00 (in 4.009507 seconds) notificationworker stdout | 2023-01-17 06:15:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:01 UTC)" (scheduled at 2023-01-17 06:15:01.636362+00:00) notificationworker stdout | 2023-01-17 06:15:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:15:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:15:01,678 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:15:01,680 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:15:01,683 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:15:01,685 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:15:01,687 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:15:02,101 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:15:02,486 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:15:02,605 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:15:02,607 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:15:03,081 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:15:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:15:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:19.346120+00:00 (in 14.408839 seconds) repositorygcworker stdout | 2023-01-17 06:15:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:04 UTC)" (scheduled at 2023-01-17 06:15:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:15:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:15:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 4, 937725), True, datetime.datetime(2023, 1, 17, 6, 15, 4, 937725), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:15:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:15:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:15:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:15:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:15.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 06:15:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:15 UTC)" (scheduled at 2023-01-17 06:15:05.646354+00:00) notificationworker stdout | 2023-01-17 06:15:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 5, 647206), True, datetime.datetime(2023, 1, 17, 6, 15, 5, 647206), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:15:05,665 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:15:05,668 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:15:05,670 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:15:06,158 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:15:06,161 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:15:06,163 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:15:06,171 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:15:06,173 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:15:06,176 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:15:06,180 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:15:06,182 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:15:06,185 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:15:06,280 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:15:06,354 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:15:07,060 [231] [DEBUG] [app] Starting request: urn:request:e4af4106-086d-4a13-841f-aa1cf759f569 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:07,060 [236] [DEBUG] [app] Starting request: urn:request:3d1a9c4c-d373-4438-a609-9d3be778642e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:07,065 [230] [DEBUG] [app] Starting request: urn:request:0f1f80b2-a715-4e73-bab5-ff78ca6a96fb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:07,066 [243] [DEBUG] [app] Starting request: urn:request:f77b0ae8-dd7b-454c-9425-c7341db57703 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:07,066 [230] [DEBUG] [app] Ending request: urn:request:0f1f80b2-a715-4e73-bab5-ff78ca6a96fb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f1f80b2-a715-4e73-bab5-ff78ca6a96fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:15:07,066 [243] [DEBUG] [app] Ending request: urn:request:f77b0ae8-dd7b-454c-9425-c7341db57703 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f77b0ae8-dd7b-454c-9425-c7341db57703', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:15:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:15:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:07,071 [233] [DEBUG] [app] Starting request: urn:request:166fab3e-ce1f-4a43-9526-a4ce5efd3bae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:07,071 [235] [DEBUG] [app] Starting request: urn:request:ee411065-9472-4a4c-9419-1e1028e7db46 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:07,071 [233] [DEBUG] [app] Ending request: urn:request:166fab3e-ce1f-4a43-9526-a4ce5efd3bae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:166fab3e-ce1f-4a43-9526-a4ce5efd3bae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:07,072 [235] [DEBUG] [app] Ending request: urn:request:ee411065-9472-4a4c-9419-1e1028e7db46 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee411065-9472-4a4c-9419-1e1028e7db46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:07,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:07,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:07,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:07,088 [236] [DEBUG] [app] Ending request: urn:request:3d1a9c4c-d373-4438-a609-9d3be778642e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d1a9c4c-d373-4438-a609-9d3be778642e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:07,088 [231] [DEBUG] [app] Ending request: urn:request:e4af4106-086d-4a13-841f-aa1cf759f569 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4af4106-086d-4a13-841f-aa1cf759f569', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 06:15:07,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 06:15:07,479 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:15:07,496 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:15:07,878 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:15:08,336 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:15:08,401 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:15:08,544 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:15:08,860 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:15:09,300 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:15:09,442 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:09,454 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:09,454 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:09,465 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:15:09,820 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:15:09,947 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:15:10,216 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:15:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:15:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:40.738568+00:00 (in 29.999493 seconds) buildlogsarchiver stdout | 2023-01-17 06:15:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:40 UTC)" (scheduled at 2023-01-17 06:15:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:15:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 15, 10, 739469), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:15:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:15:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:15:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:15:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:25.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 06:15:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:25 UTC)" (scheduled at 2023-01-17 06:15:15.646354+00:00) notificationworker stdout | 2023-01-17 06:15:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 15, 647211), True, datetime.datetime(2023, 1, 17, 6, 15, 15, 647211), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:15:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:19.537472+00:00 (in 2.599984 seconds) securityworker stdout | 2023-01-17 06:15:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:46 UTC)" (scheduled at 2023-01-17 06:15:16.936988+00:00) securityworker stdout | 2023-01-17 06:15:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:15:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:15:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:04.936835+00:00 (in 45.590248 seconds) repositorygcworker stdout | 2023-01-17 06:15:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:19 UTC)" (scheduled at 2023-01-17 06:15:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:15:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:15:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:19 UTC)" executed successfully builder stdout | 2023-01-17 06:15:19,475 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:19,487 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:19,487 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:19,498 [60] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:46.936988+00:00 (in 27.399039 seconds) securityworker stdout | 2023-01-17 06:15:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:49 UTC)" (scheduled at 2023-01-17 06:15:19.537472+00:00) securityworker stdout | 2023-01-17 06:15:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:15:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:15:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:19,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:15:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:15:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:15:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:15:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:49 UTC)" executed successfully gcworker stdout | 2023-01-17 06:15:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:15:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:51.348605+00:00 (in 29.999490 seconds) gcworker stdout | 2023-01-17 06:15:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:51 UTC)" (scheduled at 2023-01-17 06:15:21.348605+00:00) gcworker stdout | 2023-01-17 06:15:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726521349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:15:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:15:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:15:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:15:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:15:22,060 [233] [DEBUG] [app] Starting request: urn:request:32624922-ddec-4c5f-afca-0ccbdcb2a1a9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:22,060 [236] [DEBUG] [app] Starting request: urn:request:67e51921-132e-47c6-b8ff-ac6c598361f7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:22,064 [242] [DEBUG] [app] Starting request: urn:request:629074ac-e255-4c6c-bebe-c2f37eedeb20 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,064 [241] [DEBUG] [app] Starting request: urn:request:e1c0a23d-b313-4c81-b2a3-16bbfc8f584d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,065 [242] [DEBUG] [app] Ending request: urn:request:629074ac-e255-4c6c-bebe-c2f37eedeb20 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:629074ac-e255-4c6c-bebe-c2f37eedeb20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,065 [241] [DEBUG] [app] Ending request: urn:request:e1c0a23d-b313-4c81-b2a3-16bbfc8f584d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e1c0a23d-b313-4c81-b2a3-16bbfc8f584d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:15:22,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:22,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:15:22,066 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:22,068 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:22,069 [235] [DEBUG] [app] Starting request: urn:request:537250a5-1c03-4570-ae7d-a6ee027600ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:22,069 [231] [DEBUG] [app] Starting request: urn:request:e3cf0fde-48f2-4ee3-92ee-4531f7b91622 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:22,070 [235] [DEBUG] [app] Ending request: urn:request:537250a5-1c03-4570-ae7d-a6ee027600ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:537250a5-1c03-4570-ae7d-a6ee027600ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:22,070 [231] [DEBUG] [app] Ending request: urn:request:e3cf0fde-48f2-4ee3-92ee-4531f7b91622 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e3cf0fde-48f2-4ee3-92ee-4531f7b91622', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:22,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:22,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:15:22,070 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:22,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:22,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:22,087 [233] [DEBUG] [app] Ending request: urn:request:32624922-ddec-4c5f-afca-0ccbdcb2a1a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:32624922-ddec-4c5f-afca-0ccbdcb2a1a9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:22,087 [236] [DEBUG] [app] Ending request: urn:request:67e51921-132e-47c6-b8ff-ac6c598361f7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:67e51921-132e-47c6-b8ff-ac6c598361f7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:22,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:15:22,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:15:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:35.646354+00:00 (in 9.999507 seconds) notificationworker stdout | 2023-01-17 06:15:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:35 UTC)" (scheduled at 2023-01-17 06:15:25.646354+00:00) notificationworker stdout | 2023-01-17 06:15:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 25, 647239), True, datetime.datetime(2023, 1, 17, 6, 15, 25, 647239), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:15:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:15:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:35.853396+00:00 (in 6.889277 seconds) exportactionlogsworker stdout | 2023-01-17 06:15:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:28 UTC)" (scheduled at 2023-01-17 06:15:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:15:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:15:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:28 UTC)" executed successfully builder stdout | 2023-01-17 06:15:29,508 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:29,521 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:29,521 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:29,531 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:15:31,140 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:15:31,186 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:15:31,686 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:15:31,691 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:15:31,693 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:15:31,696 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:15:31,699 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:15:32,123 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:15:32,503 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:15:32,622 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:15:32,640 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:15:33,100 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:15:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:45.646354+00:00 (in 9.999526 seconds) notificationworker stdout | 2023-01-17 06:15:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:45 UTC)" (scheduled at 2023-01-17 06:15:35.646354+00:00) notificationworker stdout | 2023-01-17 06:15:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 35, 647240), True, datetime.datetime(2023, 1, 17, 6, 15, 35, 647240), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:15:35,672 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:15:35,675 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:15:35,678 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:15:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:15:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:28.963662+00:00 (in 53.109776 seconds) exportactionlogsworker stdout | 2023-01-17 06:15:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:35 UTC)" (scheduled at 2023-01-17 06:15:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:15:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:15:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 35, 854254), True, datetime.datetime(2023, 1, 17, 6, 15, 35, 854254), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:15:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:15:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:15:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:15:36,166 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:15:36,171 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:15:36,173 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:15:36,181 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:15:36,183 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:15:36,186 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:15:36,189 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:15:36,191 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:15:36,194 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:15:36,299 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:15:36,380 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:15:37,060 [231] [DEBUG] [app] Starting request: urn:request:5d1db75b-7825-4373-b46a-5f5eed544ee3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:37,060 [236] [DEBUG] [app] Starting request: urn:request:67a6c4f0-781c-415a-8957-78f1c69e44f0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:37,065 [230] [DEBUG] [app] Starting request: urn:request:ed4b6353-75d5-4f5e-bd51-b30b81deb698 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:37,066 [241] [DEBUG] [app] Starting request: urn:request:d19295de-aae4-4db5-9fcf-ae37e2798d07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:37,066 [230] [DEBUG] [app] Ending request: urn:request:ed4b6353-75d5-4f5e-bd51-b30b81deb698 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ed4b6353-75d5-4f5e-bd51-b30b81deb698', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:15:37,066 [241] [DEBUG] [app] Ending request: urn:request:d19295de-aae4-4db5-9fcf-ae37e2798d07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d19295de-aae4-4db5-9fcf-ae37e2798d07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:15:37,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:15:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:37,071 [231] [DEBUG] [app] Starting request: urn:request:100f6189-77f1-413c-ab07-9da4fff6c14d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:37,071 [231] [DEBUG] [app] Ending request: urn:request:100f6189-77f1-413c-ab07-9da4fff6c14d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:100f6189-77f1-413c-ab07-9da4fff6c14d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:37,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:37,072 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:37,073 [235] [DEBUG] [app] Starting request: urn:request:3b8f852c-fad8-46b4-a4fb-f746682f6bb0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:37,074 [235] [DEBUG] [app] Ending request: urn:request:3b8f852c-fad8-46b4-a4fb-f746682f6bb0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b8f852c-fad8-46b4-a4fb-f746682f6bb0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:37,074 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:15:37,074 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:37,075 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:37,081 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:37,088 [236] [DEBUG] [app] Ending request: urn:request:67a6c4f0-781c-415a-8957-78f1c69e44f0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:67a6c4f0-781c-415a-8957-78f1c69e44f0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:15:37,090 [231] [DEBUG] [app] Ending request: urn:request:5d1db75b-7825-4373-b46a-5f5eed544ee3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d1db75b-7825-4373-b46a-5f5eed544ee3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:37,090 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:37,091 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.031) securityscanningnotificationworker stdout | 2023-01-17 06:15:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:15:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:37.537407+00:00 (in 0.280334 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:15:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:37 UTC)" (scheduled at 2023-01-17 06:15:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:15:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:15:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 37, 257495), True, datetime.datetime(2023, 1, 17, 6, 15, 37, 257495), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:15:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:15:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:15:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:15:37,485 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:15:37,532 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} securityscanningnotificationworker stdout | 2023-01-17 06:15:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:15:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:23.537048+00:00 (in 45.999231 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:15:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:37 UTC)" (scheduled at 2023-01-17 06:15:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:15:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:15:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:37 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:15:37,902 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:15:38,371 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:15:38,405 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:15:38,580 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:15:38,868 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:15:39,327 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:15:39,537 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:39,549 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:39,549 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:39,560 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:15:39,850 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:15:39,983 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:15:40,233 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:15:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:15:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:10.738568+00:00 (in 29.999530 seconds) buildlogsarchiver stdout | 2023-01-17 06:15:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:10 UTC)" (scheduled at 2023-01-17 06:15:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:15:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 15, 40, 739491), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:15:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:15:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:15:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:15:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:15:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:42.941536+00:00 (in 1.792770 seconds) namespacegcworker stdout | 2023-01-17 06:15:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:41 UTC)" (scheduled at 2023-01-17 06:15:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:15:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:15:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 41, 149166), True, datetime.datetime(2023, 1, 17, 6, 15, 41, 149166), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:15:41,162 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:15:41,162 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:15:41,162 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:15:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:15:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:41.148245+00:00 (in 58.206222 seconds) namespacegcworker stdout | 2023-01-17 06:15:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:42 UTC)" (scheduled at 2023-01-17 06:15:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:15:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:15:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:16:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:15:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:45,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:55.646354+00:00 (in 9.999530 seconds) notificationworker stdout | 2023-01-17 06:15:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:55 UTC)" (scheduled at 2023-01-17 06:15:45.646354+00:00) notificationworker stdout | 2023-01-17 06:15:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 45, 647253), True, datetime.datetime(2023, 1, 17, 6, 15, 45, 647253), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:15:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:15:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:15:49.537472+00:00 (in 2.600030 seconds) securityworker stdout | 2023-01-17 06:15:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:16 UTC)" (scheduled at 2023-01-17 06:15:46.936988+00:00) securityworker stdout | 2023-01-17 06:15:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:46,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:15:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:15:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:16.936988+00:00 (in 27.399046 seconds) securityworker stdout | 2023-01-17 06:15:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:19 UTC)" (scheduled at 2023-01-17 06:15:49.537472+00:00) securityworker stdout | 2023-01-17 06:15:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:15:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:15:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:15:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:15:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:15:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:15:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:15:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:15:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:15:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:15:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:15:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:15:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:15:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:19 UTC)" executed successfully builder stdout | 2023-01-17 06:15:49,570 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:49,581 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:49,581 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:49,592 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:15:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:15:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:21.348605+00:00 (in 29.999478 seconds) gcworker stdout | 2023-01-17 06:15:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:21 UTC)" (scheduled at 2023-01-17 06:15:51.348605+00:00) gcworker stdout | 2023-01-17 06:15:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726551349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:15:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:15:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:15:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:15:52,060 [231] [DEBUG] [app] Starting request: urn:request:bf954fd0-80b3-4493-80c4-9cc6b5ac67b2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:52,060 [236] [DEBUG] [app] Starting request: urn:request:71e9c4eb-2133-4ce1-871f-620497d97550 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:15:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:52,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:15:52,065 [242] [DEBUG] [app] Starting request: urn:request:3b85190f-87d2-4a2c-a19c-5c2f91b4be6f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:52,065 [230] [DEBUG] [app] Starting request: urn:request:99c7fd83-9fcf-4bed-98fb-1a93a98d6582 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:15:52,066 [242] [DEBUG] [app] Ending request: urn:request:3b85190f-87d2-4a2c-a19c-5c2f91b4be6f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b85190f-87d2-4a2c-a19c-5c2f91b4be6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:15:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:15:52,066 [230] [DEBUG] [app] Ending request: urn:request:99c7fd83-9fcf-4bed-98fb-1a93a98d6582 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99c7fd83-9fcf-4bed-98fb-1a93a98d6582', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:15:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:15:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:15:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:52,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:15:52,070 [235] [DEBUG] [app] Starting request: urn:request:b61bfb21-7808-4c00-92ef-de427eeecc4c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:52,070 [233] [DEBUG] [app] Starting request: urn:request:3b010e4a-9cca-4b9d-b657-564ac13e22b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:15:52,071 [235] [DEBUG] [app] Ending request: urn:request:b61bfb21-7808-4c00-92ef-de427eeecc4c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b61bfb21-7808-4c00-92ef-de427eeecc4c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:15:52,071 [233] [DEBUG] [app] Ending request: urn:request:3b010e4a-9cca-4b9d-b657-564ac13e22b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b010e4a-9cca-4b9d-b657-564ac13e22b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:15:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:15:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:15:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:15:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:15:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:15:52,087 [231] [DEBUG] [app] Ending request: urn:request:bf954fd0-80b3-4493-80c4-9cc6b5ac67b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf954fd0-80b3-4493-80c4-9cc6b5ac67b2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:52,087 [236] [DEBUG] [app] Ending request: urn:request:71e9c4eb-2133-4ce1-871f-620497d97550 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71e9c4eb-2133-4ce1-871f-620497d97550', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:15:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:15:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:15:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:15:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:15:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:15:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:01.636362+00:00 (in 5.989487 seconds) notificationworker stdout | 2023-01-17 06:15:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:05 UTC)" (scheduled at 2023-01-17 06:15:55.646354+00:00) notificationworker stdout | 2023-01-17 06:15:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:15:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 15, 55, 647347), True, datetime.datetime(2023, 1, 17, 6, 15, 55, 647347), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:15:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:15:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:15:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:05 UTC)" executed successfully builder stdout | 2023-01-17 06:15:59,602 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:15:59,614 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:15:59,614 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:15:59,625 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:01,154 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:16:01,204 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:16:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:05.646354+00:00 (in 4.009474 seconds) notificationworker stdout | 2023-01-17 06:16:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:01 UTC)" (scheduled at 2023-01-17 06:16:01.636362+00:00) notificationworker stdout | 2023-01-17 06:16:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:16:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:16:01,695 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:16:01,698 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:16:01,700 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:16:01,703 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:16:01,705 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:16:02,151 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:16:02,538 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:16:02,646 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:16:02,648 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:16:03,106 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:16:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:16:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:17.345601+00:00 (in 12.408319 seconds) repositorygcworker stdout | 2023-01-17 06:16:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:04 UTC)" (scheduled at 2023-01-17 06:16:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:16:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:16:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 4, 937767), True, datetime.datetime(2023, 1, 17, 6, 16, 4, 937767), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:16:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:16:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:16:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:15.646354+00:00 (in 9.999520 seconds) notificationworker stdout | 2023-01-17 06:16:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:15 UTC)" (scheduled at 2023-01-17 06:16:05.646354+00:00) notificationworker stdout | 2023-01-17 06:16:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 5, 647233), True, datetime.datetime(2023, 1, 17, 6, 16, 5, 647233), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:16:05,679 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:16:05,681 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:16:05,684 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:16:06,172 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:16:06,180 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:16:06,183 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:16:06,190 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:16:06,192 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:16:06,195 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:16:06,197 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:16:06,199 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:16:06,202 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:16:06,328 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:16:06,387 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:16:07,060 [231] [DEBUG] [app] Starting request: urn:request:d4495fa9-d8e9-492d-85ce-20444f660ca7 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:07,060 [233] [DEBUG] [app] Starting request: urn:request:f4420375-d005-48b8-9ef0-b377ec96fced (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,062 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:07,065 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:07,065 [242] [DEBUG] [app] Starting request: urn:request:d362d5a6-219d-4725-88e0-f746962354d0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:07,066 [242] [DEBUG] [app] Ending request: urn:request:d362d5a6-219d-4725-88e0-f746962354d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d362d5a6-219d-4725-88e0-f746962354d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:16:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:16:07,066 [230] [DEBUG] [app] Starting request: urn:request:e41eb835-6b2c-4c88-8a22-353ddd8cbff7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:16:07,066 [230] [DEBUG] [app] Ending request: urn:request:e41eb835-6b2c-4c88-8a22-353ddd8cbff7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e41eb835-6b2c-4c88-8a22-353ddd8cbff7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:16:07,067 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:07,067 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,069 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:07,070 [236] [DEBUG] [app] Starting request: urn:request:3d290f8d-5883-424b-a833-0ab7636c885d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:07,070 [236] [DEBUG] [app] Ending request: urn:request:3d290f8d-5883-424b-a833-0ab7636c885d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d290f8d-5883-424b-a833-0ab7636c885d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:07,071 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:07,072 [235] [DEBUG] [app] Starting request: urn:request:e162f6ce-71d2-4980-9828-26cf54a31ed1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:07,072 [235] [DEBUG] [app] Ending request: urn:request:e162f6ce-71d2-4980-9828-26cf54a31ed1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e162f6ce-71d2-4980-9828-26cf54a31ed1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:07,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:16:07,073 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:07,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:07,081 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:07,087 [231] [DEBUG] [app] Ending request: urn:request:d4495fa9-d8e9-492d-85ce-20444f660ca7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d4495fa9-d8e9-492d-85ce-20444f660ca7', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:16:07,090 [233] [DEBUG] [app] Ending request: urn:request:f4420375-d005-48b8-9ef0-b377ec96fced (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f4420375-d005-48b8-9ef0-b377ec96fced', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:07,091 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:07,091 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) storagereplication stdout | 2023-01-17 06:16:07,521 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:16:07,566 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:16:07,937 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:16:08,390 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:16:08,411 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:16:08,614 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:16:08,883 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:16:09,345 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:16:09,625 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:09,638 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:09,638 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:09,649 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:16:09,856 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:16:10,004 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:16:10,268 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:16:10,739 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:16:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:40.738568+00:00 (in 29.999431 seconds) buildlogsarchiver stdout | 2023-01-17 06:16:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:40 UTC)" (scheduled at 2023-01-17 06:16:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:16:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 16, 10, 739544), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:16:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:16:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:16:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:25.646354+00:00 (in 9.999560 seconds) notificationworker stdout | 2023-01-17 06:16:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:25 UTC)" (scheduled at 2023-01-17 06:16:15.646354+00:00) notificationworker stdout | 2023-01-17 06:16:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 15, 647191), True, datetime.datetime(2023, 1, 17, 6, 16, 15, 647191), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:16:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:19.537472+00:00 (in 2.599977 seconds) securityworker stdout | 2023-01-17 06:16:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:46 UTC)" (scheduled at 2023-01-17 06:16:16.936988+00:00) securityworker stdout | 2023-01-17 06:16:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:16:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:16:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:19.346120+00:00 (in 2.000021 seconds) repositorygcworker stdout | 2023-01-17 06:16:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:17 UTC)" (scheduled at 2023-01-17 06:16:17.345601+00:00) repositorygcworker stdout | 2023-01-17 06:16:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 16, 17, 346472), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:16:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 17, 346472), True, datetime.datetime(2023, 1, 17, 6, 16, 17, 346472), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:16:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 17, 346472), True, datetime.datetime(2023, 1, 17, 6, 16, 17, 346472), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 16, 17, 346472), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:16:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:16:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:16:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:16:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:04.936835+00:00 (in 45.590278 seconds) repositorygcworker stdout | 2023-01-17 06:16:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:19 UTC)" (scheduled at 2023-01-17 06:16:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:16:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:16:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:16:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:46.936988+00:00 (in 27.399023 seconds) securityworker stdout | 2023-01-17 06:16:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:49 UTC)" (scheduled at 2023-01-17 06:16:19.537472+00:00) securityworker stdout | 2023-01-17 06:16:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:16:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:16:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:16:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:16:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:16:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:16:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:49 UTC)" executed successfully builder stdout | 2023-01-17 06:16:19,659 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:19,671 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:19,671 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:19,682 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:16:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:51.348605+00:00 (in 29.999477 seconds) gcworker stdout | 2023-01-17 06:16:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:51 UTC)" (scheduled at 2023-01-17 06:16:21.348605+00:00) gcworker stdout | 2023-01-17 06:16:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726581349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:16:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:16:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:16:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:16:22,059 [235] [DEBUG] [app] Starting request: urn:request:8df6559c-f770-46d6-88dd-affb75b3ee7b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:22,060 [233] [DEBUG] [app] Starting request: urn:request:20ec0ff1-2267-4b6d-8a3d-6680c9c954a5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:22,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:22,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:22,064 [243] [DEBUG] [app] Starting request: urn:request:9307425d-abf9-4723-ac8f-be9318844753 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:22,065 [242] [DEBUG] [app] Starting request: urn:request:9820d6ea-afab-4c41-aa88-7809542157b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:22,065 [243] [DEBUG] [app] Ending request: urn:request:9307425d-abf9-4723-ac8f-be9318844753 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9307425d-abf9-4723-ac8f-be9318844753', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:22,065 [242] [DEBUG] [app] Ending request: urn:request:9820d6ea-afab-4c41-aa88-7809542157b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9820d6ea-afab-4c41-aa88-7809542157b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:16:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:16:22,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:22,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:16:22,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:22,068 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:22,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:22,070 [236] [DEBUG] [app] Starting request: urn:request:7e6f8dbf-15a1-48c7-b009-fd642682c225 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:22,070 [233] [DEBUG] [app] Starting request: urn:request:36e13b32-8f40-4bd2-a1e7-a5a7d05c3762 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:22,070 [236] [DEBUG] [app] Ending request: urn:request:7e6f8dbf-15a1-48c7-b009-fd642682c225 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e6f8dbf-15a1-48c7-b009-fd642682c225', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:22,070 [233] [DEBUG] [app] Ending request: urn:request:36e13b32-8f40-4bd2-a1e7-a5a7d05c3762 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36e13b32-8f40-4bd2-a1e7-a5a7d05c3762', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:16:22,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:22,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:22,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:22,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:22,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:22,087 [233] [DEBUG] [app] Ending request: urn:request:20ec0ff1-2267-4b6d-8a3d-6680c9c954a5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20ec0ff1-2267-4b6d-8a3d-6680c9c954a5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:22,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:22,087 [235] [DEBUG] [app] Ending request: urn:request:8df6559c-f770-46d6-88dd-affb75b3ee7b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8df6559c-f770-46d6-88dd-affb75b3ee7b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:16:22,088 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 06:16:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" securityscanningnotificationworker stdout | 2023-01-17 06:16:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:16:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:37.256652+00:00 (in 13.719165 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:16:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:23 UTC)" (scheduled at 2023-01-17 06:16:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:16:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 16, 23, 537870), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:16:23,549 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 23, 537870), True, datetime.datetime(2023, 1, 17, 6, 16, 23, 537870), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:16:23,553 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 23, 537870), True, datetime.datetime(2023, 1, 17, 6, 16, 23, 537870), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 16, 23, 537870), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:16:23,557 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:16:23,557 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:35.646354+00:00 (in 9.999561 seconds) notificationworker stdout | 2023-01-17 06:16:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:35 UTC)" (scheduled at 2023-01-17 06:16:25.646354+00:00) notificationworker stdout | 2023-01-17 06:16:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 25, 647199), True, datetime.datetime(2023, 1, 17, 6, 16, 25, 647199), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:16:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:16:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:35.853396+00:00 (in 6.889263 seconds) exportactionlogsworker stdout | 2023-01-17 06:16:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:28 UTC)" (scheduled at 2023-01-17 06:16:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:16:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:16:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:28 UTC)" executed successfully builder stdout | 2023-01-17 06:16:29,692 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:29,705 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:29,705 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:29,717 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:31,187 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:16:31,236 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:16:31,703 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:16:31,707 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:16:31,709 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:16:31,712 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:16:31,714 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:16:32,186 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:16:32,573 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:16:32,682 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:16:32,684 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:16:33,141 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:16:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:45.646354+00:00 (in 9.999515 seconds) notificationworker stdout | 2023-01-17 06:16:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:45 UTC)" (scheduled at 2023-01-17 06:16:35.646354+00:00) notificationworker stdout | 2023-01-17 06:16:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 35, 647269), True, datetime.datetime(2023, 1, 17, 6, 16, 35, 647269), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:16:35,686 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:16:35,689 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:16:35,691 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:16:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:16:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:54.963413+00:00 (in 19.109518 seconds) exportactionlogsworker stdout | 2023-01-17 06:16:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:35 UTC)" (scheduled at 2023-01-17 06:16:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:16:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:16:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 35, 854249), True, datetime.datetime(2023, 1, 17, 6, 16, 35, 854249), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:16:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:16:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:16:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:16:36,178 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:16:36,190 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:16:36,193 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:16:36,199 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:16:36,201 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:16:36,205 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:16:36,207 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:16:36,209 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:16:36,212 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:16:36,363 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:16:36,422 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:16:37,059 [235] [DEBUG] [app] Starting request: urn:request:8d544670-29c0-4caa-b2d0-70698242384b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:37,060 [233] [DEBUG] [app] Starting request: urn:request:0a482616-cb8b-4049-84aa-c1440f204d72 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:37,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:37,065 [241] [DEBUG] [app] Starting request: urn:request:df311266-cbaa-44ea-bf95-609f2a93a945 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:37,065 [243] [DEBUG] [app] Starting request: urn:request:ddabb51a-4238-4daf-90a7-a37f3c79680e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:37,065 [241] [DEBUG] [app] Ending request: urn:request:df311266-cbaa-44ea-bf95-609f2a93a945 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df311266-cbaa-44ea-bf95-609f2a93a945', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:37,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:16:37,065 [243] [DEBUG] [app] Ending request: urn:request:ddabb51a-4238-4daf-90a7-a37f3c79680e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ddabb51a-4238-4daf-90a7-a37f3c79680e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:16:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:16:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:37,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:37,069 [236] [DEBUG] [app] Starting request: urn:request:689eaadd-9579-4bff-81d8-e7bed44a90b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:37,070 [236] [DEBUG] [app] Ending request: urn:request:689eaadd-9579-4bff-81d8-e7bed44a90b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:689eaadd-9579-4bff-81d8-e7bed44a90b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:37,070 [233] [DEBUG] [app] Starting request: urn:request:94d82542-180c-4779-9c58-2121cc011338 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:37,070 [233] [DEBUG] [app] Ending request: urn:request:94d82542-180c-4779-9c58-2121cc011338 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:94d82542-180c-4779-9c58-2121cc011338', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:16:37,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:37,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:37,087 [233] [DEBUG] [app] Ending request: urn:request:0a482616-cb8b-4049-84aa-c1440f204d72 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0a482616-cb8b-4049-84aa-c1440f204d72', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:37,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:37,087 [235] [DEBUG] [app] Ending request: urn:request:8d544670-29c0-4caa-b2d0-70698242384b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d544670-29c0-4caa-b2d0-70698242384b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:37,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:16:37,087 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:16:37,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 06:16:37,257 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:16:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:37.537407+00:00 (in 0.280286 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:16:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:37 UTC)" (scheduled at 2023-01-17 06:16:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:16:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:16:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 37, 257559), True, datetime.datetime(2023, 1, 17, 6, 16, 37, 257559), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:16:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:16:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:16:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:16:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:16:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:37.256652+00:00 (in 59.718823 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:16:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:37 UTC)" (scheduled at 2023-01-17 06:16:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:16:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:16:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:16:37,549 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:16:37,589 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:16:37,952 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} manifestbackfillworker stdout | 2023-01-17 06:16:38,419 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} chunkcleanupworker stdout | 2023-01-17 06:16:38,424 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} securityscanningnotificationworker stdout | 2023-01-17 06:16:38,645 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:16:38,902 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:16:39,381 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:16:39,727 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:39,739 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:39,739 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:39,750 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:16:39,887 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:16:40,035 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:16:40,295 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:16:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:16:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:10.738568+00:00 (in 29.999529 seconds) buildlogsarchiver stdout | 2023-01-17 06:16:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:10 UTC)" (scheduled at 2023-01-17 06:16:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:16:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 16, 40, 739435), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:16:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:16:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:16:40,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:16:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:16:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:42.941536+00:00 (in 1.792802 seconds) namespacegcworker stdout | 2023-01-17 06:16:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:41 UTC)" (scheduled at 2023-01-17 06:16:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:16:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:16:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 41, 149154), True, datetime.datetime(2023, 1, 17, 6, 16, 41, 149154), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:16:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:16:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:16:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:16:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:16:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:06.941185+00:00 (in 23.999167 seconds) namespacegcworker stdout | 2023-01-17 06:16:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:42 UTC)" (scheduled at 2023-01-17 06:16:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:16:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:16:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:17:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:55.646354+00:00 (in 9.999525 seconds) notificationworker stdout | 2023-01-17 06:16:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:55 UTC)" (scheduled at 2023-01-17 06:16:45.646354+00:00) notificationworker stdout | 2023-01-17 06:16:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 45, 647190), True, datetime.datetime(2023, 1, 17, 6, 16, 45, 647190), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:16:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:16:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:16:49.537472+00:00 (in 2.600026 seconds) securityworker stdout | 2023-01-17 06:16:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:16 UTC)" (scheduled at 2023-01-17 06:16:46.936988+00:00) securityworker stdout | 2023-01-17 06:16:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:16:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:16:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:16.936988+00:00 (in 27.399042 seconds) securityworker stdout | 2023-01-17 06:16:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:19 UTC)" (scheduled at 2023-01-17 06:16:49.537472+00:00) securityworker stdout | 2023-01-17 06:16:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:16:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:16:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:16:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:16:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:16:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:16:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:16:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:16:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:16:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:16:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:16:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:16:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:16:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:19 UTC)" executed successfully builder stdout | 2023-01-17 06:16:49,760 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:49,772 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:49,772 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:49,783 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:16:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:21.348605+00:00 (in 29.999505 seconds) gcworker stdout | 2023-01-17 06:16:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:21 UTC)" (scheduled at 2023-01-17 06:16:51.348605+00:00) gcworker stdout | 2023-01-17 06:16:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726611349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:16:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:16:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:16:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:16:52,060 [235] [DEBUG] [app] Starting request: urn:request:557dc7a2-39c0-43ec-a512-20235009c004 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:52,060 [231] [DEBUG] [app] Starting request: urn:request:b08a237b-6048-4ceb-90db-bab016334614 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:16:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:16:52,065 [230] [DEBUG] [app] Starting request: urn:request:53cee8c8-bddf-4aea-b254-c3a6bd214f59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:52,065 [241] [DEBUG] [app] Starting request: urn:request:5c97d2a6-6eaf-4263-b9fe-680a7e9b999c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:16:52,066 [230] [DEBUG] [app] Ending request: urn:request:53cee8c8-bddf-4aea-b254-c3a6bd214f59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:53cee8c8-bddf-4aea-b254-c3a6bd214f59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:52,066 [241] [DEBUG] [app] Ending request: urn:request:5c97d2a6-6eaf-4263-b9fe-680a7e9b999c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c97d2a6-6eaf-4263-b9fe-680a7e9b999c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:16:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:16:52,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:16:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:16:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:16:52,070 [233] [DEBUG] [app] Starting request: urn:request:ed68b807-b0a0-4d3e-ac9f-ecc2e9ed0171 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:52,070 [236] [DEBUG] [app] Starting request: urn:request:2aee13e5-b612-4cd0-8f21-e6393279357f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:16:52,071 [233] [DEBUG] [app] Ending request: urn:request:ed68b807-b0a0-4d3e-ac9f-ecc2e9ed0171 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ed68b807-b0a0-4d3e-ac9f-ecc2e9ed0171', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:52,071 [236] [DEBUG] [app] Ending request: urn:request:2aee13e5-b612-4cd0-8f21-e6393279357f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2aee13e5-b612-4cd0-8f21-e6393279357f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:16:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:16:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:16:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:16:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:16:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:16:52,087 [235] [DEBUG] [app] Ending request: urn:request:557dc7a2-39c0-43ec-a512-20235009c004 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:557dc7a2-39c0-43ec-a512-20235009c004', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:52,087 [231] [DEBUG] [app] Ending request: urn:request:b08a237b-6048-4ceb-90db-bab016334614 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b08a237b-6048-4ceb-90db-bab016334614', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:16:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:16:52,087 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:16:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:16:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:16:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" exportactionlogsworker stdout | 2023-01-17 06:16:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:16:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:28.963662+00:00 (in 33.999823 seconds) exportactionlogsworker stdout | 2023-01-17 06:16:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:54 UTC)" (scheduled at 2023-01-17 06:16:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 06:16:54,965 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 16, 54, 964335), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:16:54,976 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 54, 964335), True, datetime.datetime(2023, 1, 17, 6, 16, 54, 964335), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:16:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 16, 54, 964335), True, datetime.datetime(2023, 1, 17, 6, 16, 54, 964335), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 16, 54, 964335), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:16:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:16:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:21:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:16:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:16:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:01.636362+00:00 (in 5.989586 seconds) notificationworker stdout | 2023-01-17 06:16:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:05 UTC)" (scheduled at 2023-01-17 06:16:55.646354+00:00) notificationworker stdout | 2023-01-17 06:16:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:16:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 16, 55, 647193), True, datetime.datetime(2023, 1, 17, 6, 16, 55, 647193), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:16:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:16:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:16:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:05 UTC)" executed successfully builder stdout | 2023-01-17 06:16:59,793 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:16:59,805 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:16:59,806 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:16:59,816 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:01,215 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:17:01,241 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:17:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:05.646354+00:00 (in 4.009492 seconds) notificationworker stdout | 2023-01-17 06:17:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:01 UTC)" (scheduled at 2023-01-17 06:17:01.636362+00:00) notificationworker stdout | 2023-01-17 06:17:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:17:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:17:01,710 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:17:01,713 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:17:01,715 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:17:01,718 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:17:01,720 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:17:02,218 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:17:02,609 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:17:02,698 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:17:02,714 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:17:03,150 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:17:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:17:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:19.346120+00:00 (in 14.408840 seconds) repositorygcworker stdout | 2023-01-17 06:17:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:04 UTC)" (scheduled at 2023-01-17 06:17:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:17:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:17:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 4, 937742), True, datetime.datetime(2023, 1, 17, 6, 17, 4, 937742), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:17:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:17:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:17:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:17:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:15.646354+00:00 (in 9.999489 seconds) notificationworker stdout | 2023-01-17 06:17:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:15 UTC)" (scheduled at 2023-01-17 06:17:05.646354+00:00) notificationworker stdout | 2023-01-17 06:17:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 5, 647279), True, datetime.datetime(2023, 1, 17, 6, 17, 5, 647279), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:17:05,694 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:17:05,696 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:17:05,699 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:17:06,184 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:17:06,199 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:17:06,202 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:17:06,206 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:17:06,208 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:17:06,212 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:17:06,216 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:17:06,218 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-registry stdout | 2023-01-17 06:17:06,220 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} buildlogsarchiver stdout | 2023-01-17 06:17:06,379 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:17:06,454 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} namespacegcworker stdout | 2023-01-17 06:17:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:17:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:41.148245+00:00 (in 34.206579 seconds) namespacegcworker stdout | 2023-01-17 06:17:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:22:06 UTC)" (scheduled at 2023-01-17 06:17:06.941185+00:00) namespacegcworker stdout | 2023-01-17 06:17:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 17, 6, 942067), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:17:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 17, 6, 942067), True, datetime.datetime(2023, 1, 17, 6, 17, 6, 942067), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:17:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 17, 6, 942067), True, datetime.datetime(2023, 1, 17, 6, 17, 6, 942067), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 17, 6, 942067), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:17:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:17:06,961 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:22:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:17:07,060 [235] [DEBUG] [app] Starting request: urn:request:ea3306a5-db05-41ab-9944-0a953c7922a2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:07,060 [233] [DEBUG] [app] Starting request: urn:request:4da7f75b-807c-4d12-b7bb-dcb9708e5836 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:07,065 [230] [DEBUG] [app] Starting request: urn:request:08cc648b-d7f9-4724-87a3-583b3e4a9e34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:07,065 [230] [DEBUG] [app] Ending request: urn:request:08cc648b-d7f9-4724-87a3-583b3e4a9e34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08cc648b-d7f9-4724-87a3-583b3e4a9e34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:07,065 [242] [DEBUG] [app] Starting request: urn:request:d3af36d3-c41e-4532-9054-14a40d8dd3a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:17:07,066 [242] [DEBUG] [app] Ending request: urn:request:d3af36d3-c41e-4532-9054-14a40d8dd3a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d3af36d3-c41e-4532-9054-14a40d8dd3a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:17:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:17:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:07,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:07,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:07,070 [231] [DEBUG] [app] Starting request: urn:request:afac0a5f-7056-498d-924f-d7f6b23264ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:07,071 [231] [DEBUG] [app] Ending request: urn:request:afac0a5f-7056-498d-924f-d7f6b23264ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:afac0a5f-7056-498d-924f-d7f6b23264ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:07,071 [236] [DEBUG] [app] Starting request: urn:request:98d5f7ac-9371-4eff-bfd4-a326e28e8f42 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:07,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:17:07,071 [236] [DEBUG] [app] Ending request: urn:request:98d5f7ac-9371-4eff-bfd4-a326e28e8f42 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98d5f7ac-9371-4eff-bfd4-a326e28e8f42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:07,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:07,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:17:07,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:07,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:07,087 [235] [DEBUG] [app] Ending request: urn:request:ea3306a5-db05-41ab-9944-0a953c7922a2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ea3306a5-db05-41ab-9944-0a953c7922a2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:07,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:07,087 [233] [DEBUG] [app] Ending request: urn:request:4da7f75b-807c-4d12-b7bb-dcb9708e5836 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4da7f75b-807c-4d12-b7bb-dcb9708e5836', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:07,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:17:07,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:07,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) storagereplication stdout | 2023-01-17 06:17:07,583 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:17:07,612 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:17:07,987 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} manifestbackfillworker stdout | 2023-01-17 06:17:08,453 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} chunkcleanupworker stdout | 2023-01-17 06:17:08,455 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} securityscanningnotificationworker stdout | 2023-01-17 06:17:08,681 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:17:08,937 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:17:09,417 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:17:09,822 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:09,835 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:09,835 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:09,846 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:17:09,923 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:17:10,059 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:17:10,331 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:17:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:17:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:40.738568+00:00 (in 29.999529 seconds) buildlogsarchiver stdout | 2023-01-17 06:17:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:40 UTC)" (scheduled at 2023-01-17 06:17:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:17:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 17, 10, 739454), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:17:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:17:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:17:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:17:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:25.646354+00:00 (in 9.999506 seconds) notificationworker stdout | 2023-01-17 06:17:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:25 UTC)" (scheduled at 2023-01-17 06:17:15.646354+00:00) notificationworker stdout | 2023-01-17 06:17:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 15, 647245), True, datetime.datetime(2023, 1, 17, 6, 17, 15, 647245), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:17:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:19.537472+00:00 (in 2.600011 seconds) securityworker stdout | 2023-01-17 06:17:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:46 UTC)" (scheduled at 2023-01-17 06:17:16.936988+00:00) securityworker stdout | 2023-01-17 06:17:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:17:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:17:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:04.936835+00:00 (in 45.590232 seconds) repositorygcworker stdout | 2023-01-17 06:17:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:19 UTC)" (scheduled at 2023-01-17 06:17:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:17:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:17:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:17:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:46.936988+00:00 (in 27.399091 seconds) securityworker stdout | 2023-01-17 06:17:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:49 UTC)" (scheduled at 2023-01-17 06:17:19.537472+00:00) securityworker stdout | 2023-01-17 06:17:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:17:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:17:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:17:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:17:19,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:17:19,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:17:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:49 UTC)" executed successfully builder stdout | 2023-01-17 06:17:19,856 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:19,868 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:19,868 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:19,878 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:17:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:51.348605+00:00 (in 29.999477 seconds) gcworker stdout | 2023-01-17 06:17:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:51 UTC)" (scheduled at 2023-01-17 06:17:21.348605+00:00) gcworker stdout | 2023-01-17 06:17:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726641349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:17:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:17:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:17:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:17:22,060 [231] [DEBUG] [app] Starting request: urn:request:24e6acd9-ba48-4d65-821b-d432eead3a95 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:22,060 [236] [DEBUG] [app] Starting request: urn:request:728d3c06-d15d-43a2-b434-48bc3d93f000 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:22,066 [230] [DEBUG] [app] Starting request: urn:request:dcf84328-6829-41e3-9822-06b859c4d9a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:22,066 [242] [DEBUG] [app] Starting request: urn:request:461060ed-21b4-4027-9bb1-a4d4af2e7c5f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:22,066 [230] [DEBUG] [app] Ending request: urn:request:dcf84328-6829-41e3-9822-06b859c4d9a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dcf84328-6829-41e3-9822-06b859c4d9a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:17:22,067 [242] [DEBUG] [app] Ending request: urn:request:461060ed-21b4-4027-9bb1-a4d4af2e7c5f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:461060ed-21b4-4027-9bb1-a4d4af2e7c5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:22,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:17:22,067 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:17:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:22,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:22,071 [235] [DEBUG] [app] Starting request: urn:request:821ef466-5805-4817-9ba2-77866728edc2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:22,071 [233] [DEBUG] [app] Starting request: urn:request:97c64953-5610-48b7-beda-d83526a4bbcb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:22,071 [235] [DEBUG] [app] Ending request: urn:request:821ef466-5805-4817-9ba2-77866728edc2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:821ef466-5805-4817-9ba2-77866728edc2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:22,071 [233] [DEBUG] [app] Ending request: urn:request:97c64953-5610-48b7-beda-d83526a4bbcb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97c64953-5610-48b7-beda-d83526a4bbcb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:22,072 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:22,088 [231] [DEBUG] [app] Ending request: urn:request:24e6acd9-ba48-4d65-821b-d432eead3a95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:24e6acd9-ba48-4d65-821b-d432eead3a95', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:22,088 [236] [DEBUG] [app] Ending request: urn:request:728d3c06-d15d-43a2-b434-48bc3d93f000 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:728d3c06-d15d-43a2-b434-48bc3d93f000', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:17:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:17:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:35.646354+00:00 (in 9.999548 seconds) notificationworker stdout | 2023-01-17 06:17:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:35 UTC)" (scheduled at 2023-01-17 06:17:25.646354+00:00) notificationworker stdout | 2023-01-17 06:17:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 25, 647203), True, datetime.datetime(2023, 1, 17, 6, 17, 25, 647203), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:17:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:17:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:35.853396+00:00 (in 6.889207 seconds) exportactionlogsworker stdout | 2023-01-17 06:17:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:28 UTC)" (scheduled at 2023-01-17 06:17:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:17:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:17:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:28 UTC)" executed successfully builder stdout | 2023-01-17 06:17:29,881 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:29,893 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:29,894 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:29,904 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:31,234 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:17:31,276 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:17:31,718 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:17:31,721 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:17:31,725 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:17:31,727 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:17:31,730 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:17:32,228 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:17:32,638 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:17:32,709 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:17:32,749 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:17:33,185 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:17:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:45.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 06:17:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:45 UTC)" (scheduled at 2023-01-17 06:17:35.646354+00:00) notificationworker stdout | 2023-01-17 06:17:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 35, 647203), True, datetime.datetime(2023, 1, 17, 6, 17, 35, 647203), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:35,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:17:35,701 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:17:35,704 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:17:35,706 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:17:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:17:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:28.963662+00:00 (in 53.109741 seconds) exportactionlogsworker stdout | 2023-01-17 06:17:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:35 UTC)" (scheduled at 2023-01-17 06:17:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:17:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:17:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 35, 854391), True, datetime.datetime(2023, 1, 17, 6, 17, 35, 854391), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:17:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:17:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:17:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:17:36,193 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:17:36,207 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:17:36,209 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:17:36,213 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:17:36,215 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:17:36,218 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:17:36,222 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:17:36,226 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:17:36,228 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:17:36,414 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:17:36,486 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:17:37,059 [235] [DEBUG] [app] Starting request: urn:request:b7be5b41-42ef-4ee0-b674-d95d54e5884e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:37,060 [236] [DEBUG] [app] Starting request: urn:request:809d258d-2cde-47ab-a4f8-803b7a10cbc6 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:37,064 [230] [DEBUG] [app] Starting request: urn:request:71dd3bb9-556c-406e-bb33-1c3e9a155db5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:37,065 [243] [DEBUG] [app] Starting request: urn:request:c1e8ba87-44d1-4e98-9a02-6743b0964214 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:37,065 [230] [DEBUG] [app] Ending request: urn:request:71dd3bb9-556c-406e-bb33-1c3e9a155db5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71dd3bb9-556c-406e-bb33-1c3e9a155db5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:17:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:17:37,065 [243] [DEBUG] [app] Ending request: urn:request:c1e8ba87-44d1-4e98-9a02-6743b0964214 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c1e8ba87-44d1-4e98-9a02-6743b0964214', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:37,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:17:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:17:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:37,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:37,070 [233] [DEBUG] [app] Starting request: urn:request:00e41576-a55f-4415-8c6b-9ddebce2d241 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:37,071 [233] [DEBUG] [app] Ending request: urn:request:00e41576-a55f-4415-8c6b-9ddebce2d241 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00e41576-a55f-4415-8c6b-9ddebce2d241', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:37,071 [231] [DEBUG] [app] Starting request: urn:request:befcdbee-95ec-4685-a76c-e93e0a1ebf06 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:17:37,071 [231] [DEBUG] [app] Ending request: urn:request:befcdbee-95ec-4685-a76c-e93e0a1ebf06 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:befcdbee-95ec-4685-a76c-e93e0a1ebf06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:37,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:37,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:37,088 [235] [DEBUG] [app] Ending request: urn:request:b7be5b41-42ef-4ee0-b674-d95d54e5884e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b7be5b41-42ef-4ee0-b674-d95d54e5884e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:37,088 [236] [DEBUG] [app] Ending request: urn:request:809d258d-2cde-47ab-a4f8-803b7a10cbc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:809d258d-2cde-47ab-a4f8-803b7a10cbc6', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:17:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 06:17:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:17:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:37.537407+00:00 (in 0.280356 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:17:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:37 UTC)" (scheduled at 2023-01-17 06:17:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:17:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:17:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 37, 257476), True, datetime.datetime(2023, 1, 17, 6, 17, 37, 257476), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:17:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:17:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:17:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:17:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:17:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:37.256652+00:00 (in 59.718819 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:17:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:37 UTC)" (scheduled at 2023-01-17 06:17:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:17:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:17:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:17:37,601 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:17:37,623 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:17:38,023 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} manifestbackfillworker stdout | 2023-01-17 06:17:38,487 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} chunkcleanupworker stdout | 2023-01-17 06:17:38,490 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} securityscanningnotificationworker stdout | 2023-01-17 06:17:38,716 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:17:38,962 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:17:39,441 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} builder stdout | 2023-01-17 06:17:39,914 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:39,927 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:39,927 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:39,938 [60] [DEBUG] [data.database] Disconnecting from database. teamsyncworker stdout | 2023-01-17 06:17:39,950 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:17:40,090 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:17:40,344 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:17:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:17:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:10.738568+00:00 (in 29.999516 seconds) buildlogsarchiver stdout | 2023-01-17 06:17:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:10 UTC)" (scheduled at 2023-01-17 06:17:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:17:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 17, 40, 739464), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:17:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:17:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:17:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:17:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:17:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:42.941536+00:00 (in 1.792806 seconds) namespacegcworker stdout | 2023-01-17 06:17:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:41 UTC)" (scheduled at 2023-01-17 06:17:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:17:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:17:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 41, 149129), True, datetime.datetime(2023, 1, 17, 6, 17, 41, 149129), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:17:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:17:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:17:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:17:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:17:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:41.148245+00:00 (in 58.206242 seconds) namespacegcworker stdout | 2023-01-17 06:17:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:42 UTC)" (scheduled at 2023-01-17 06:17:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:17:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:17:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:18:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:17:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:55.646354+00:00 (in 9.999527 seconds) notificationworker stdout | 2023-01-17 06:17:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:55 UTC)" (scheduled at 2023-01-17 06:17:45.646354+00:00) notificationworker stdout | 2023-01-17 06:17:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 45, 647222), True, datetime.datetime(2023, 1, 17, 6, 17, 45, 647222), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:45,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:17:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:17:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:17:49.537472+00:00 (in 2.600009 seconds) securityworker stdout | 2023-01-17 06:17:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:16 UTC)" (scheduled at 2023-01-17 06:17:46.936988+00:00) securityworker stdout | 2023-01-17 06:17:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:17:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:17:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:16.936988+00:00 (in 27.399017 seconds) securityworker stdout | 2023-01-17 06:17:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:19 UTC)" (scheduled at 2023-01-17 06:17:49.537472+00:00) securityworker stdout | 2023-01-17 06:17:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:17:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:17:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:17:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:17:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:17:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:17:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:17:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:17:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:17:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:17:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:17:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:17:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:17:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:19 UTC)" executed successfully builder stdout | 2023-01-17 06:17:49,948 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:49,960 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:49,960 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:17:49,971 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:51,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:17:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:21.348605+00:00 (in 29.999497 seconds) gcworker stdout | 2023-01-17 06:17:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:21 UTC)" (scheduled at 2023-01-17 06:17:51.348605+00:00) gcworker stdout | 2023-01-17 06:17:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726671349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:17:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:17:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:17:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:17:52,060 [231] [DEBUG] [app] Starting request: urn:request:55188d9e-d2d6-430a-8ba3-a2246c7ac15d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:52,060 [236] [DEBUG] [app] Starting request: urn:request:38bebbc9-b7ec-4faa-a17e-11a05801da7f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:17:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:52,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:17:52,065 [230] [DEBUG] [app] Starting request: urn:request:906e2e9b-3a8d-4efd-a666-1bca5fa4eda7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:52,065 [242] [DEBUG] [app] Starting request: urn:request:438416e2-9e89-40e8-9e81-f999d7190497 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:17:52,065 [230] [DEBUG] [app] Ending request: urn:request:906e2e9b-3a8d-4efd-a666-1bca5fa4eda7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:906e2e9b-3a8d-4efd-a666-1bca5fa4eda7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:17:52,065 [242] [DEBUG] [app] Ending request: urn:request:438416e2-9e89-40e8-9e81-f999d7190497 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:438416e2-9e89-40e8-9e81-f999d7190497', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:17:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:17:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:17:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:17:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:17:52,070 [236] [DEBUG] [app] Starting request: urn:request:3bd757c4-0d23-44c0-b610-012283c50289 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:52,070 [236] [DEBUG] [app] Ending request: urn:request:3bd757c4-0d23-44c0-b610-012283c50289 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3bd757c4-0d23-44c0-b610-012283c50289', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:17:52,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:17:52,070 [235] [DEBUG] [app] Starting request: urn:request:d12c240a-75ce-4db1-abb9-2d3ad287150f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:17:52,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:52,071 [235] [DEBUG] [app] Ending request: urn:request:d12c240a-75ce-4db1-abb9-2d3ad287150f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d12c240a-75ce-4db1-abb9-2d3ad287150f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:17:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:17:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:17:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:17:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:17:52,087 [236] [DEBUG] [app] Ending request: urn:request:38bebbc9-b7ec-4faa-a17e-11a05801da7f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38bebbc9-b7ec-4faa-a17e-11a05801da7f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:52,087 [231] [DEBUG] [app] Ending request: urn:request:55188d9e-d2d6-430a-8ba3-a2246c7ac15d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55188d9e-d2d6-430a-8ba3-a2246c7ac15d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:17:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:17:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:17:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:17:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) notificationworker stdout | 2023-01-17 06:17:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:17:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:01.636362+00:00 (in 5.989529 seconds) notificationworker stdout | 2023-01-17 06:17:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:05 UTC)" (scheduled at 2023-01-17 06:17:55.646354+00:00) notificationworker stdout | 2023-01-17 06:17:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:17:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 17, 55, 647248), True, datetime.datetime(2023, 1, 17, 6, 17, 55, 647248), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:17:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:17:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:17:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:05 UTC)" executed successfully builder stdout | 2023-01-17 06:17:59,981 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:17:59,994 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:17:59,994 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:00,004 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:01,241 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:18:01,311 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:18:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:05.646354+00:00 (in 4.009561 seconds) notificationworker stdout | 2023-01-17 06:18:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:01 UTC)" (scheduled at 2023-01-17 06:18:01.636362+00:00) notificationworker stdout | 2023-01-17 06:18:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:18:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:01,725 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:18:01,728 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:18:01,731 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:18:01,733 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:18:01,736 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:18:02,237 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:18:02,651 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:18:02,740 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:18:02,753 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:18:03,206 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:18:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:18:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:19.346120+00:00 (in 14.408819 seconds) repositorygcworker stdout | 2023-01-17 06:18:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:04 UTC)" (scheduled at 2023-01-17 06:18:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:18:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:18:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 4, 937749), True, datetime.datetime(2023, 1, 17, 6, 18, 4, 937749), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:18:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:18:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:18:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:18:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:15.646354+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 06:18:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:15 UTC)" (scheduled at 2023-01-17 06:18:05.646354+00:00) notificationworker stdout | 2023-01-17 06:18:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 5, 647207), True, datetime.datetime(2023, 1, 17, 6, 18, 5, 647207), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:18:05,708 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:18:05,710 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:18:05,712 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:18:06,202 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:18:06,213 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:18:06,217 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:18:06,221 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:18:06,224 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:18:06,227 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:18:06,229 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:18:06,237 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:18:06,239 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:18:06,422 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:18:06,521 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:18:07,060 [236] [DEBUG] [app] Starting request: urn:request:3abc7393-c6a9-4443-b05b-c4fe415ea9d9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:07,060 [231] [DEBUG] [app] Starting request: urn:request:755e693c-8de3-4860-b02e-7523311efb9d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:07,065 [230] [DEBUG] [app] Starting request: urn:request:a0d86e9a-bec1-4ad9-b1d1-ca530cece4de (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:07,065 [239] [DEBUG] [app] Starting request: urn:request:3af20837-6f35-431a-87c4-38b86bd9f923 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:07,065 [230] [DEBUG] [app] Ending request: urn:request:a0d86e9a-bec1-4ad9-b1d1-ca530cece4de (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0d86e9a-bec1-4ad9-b1d1-ca530cece4de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:18:07,065 [239] [DEBUG] [app] Ending request: urn:request:3af20837-6f35-431a-87c4-38b86bd9f923 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3af20837-6f35-431a-87c4-38b86bd9f923', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:18:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:18:07,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:07,070 [231] [DEBUG] [app] Starting request: urn:request:2be68cc7-f451-483d-a578-0b5c5f695ab7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:07,070 [231] [DEBUG] [app] Ending request: urn:request:2be68cc7-f451-483d-a578-0b5c5f695ab7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2be68cc7-f451-483d-a578-0b5c5f695ab7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:07,071 [235] [DEBUG] [app] Starting request: urn:request:4d5b358f-9b96-4b74-bfe7-6eb70e60aee7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:07,071 [235] [DEBUG] [app] Ending request: urn:request:4d5b358f-9b96-4b74-bfe7-6eb70e60aee7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d5b358f-9b96-4b74-bfe7-6eb70e60aee7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:07,087 [231] [DEBUG] [app] Ending request: urn:request:755e693c-8de3-4860-b02e-7523311efb9d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:755e693c-8de3-4860-b02e-7523311efb9d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:07,087 [236] [DEBUG] [app] Ending request: urn:request:3abc7393-c6a9-4443-b05b-c4fe415ea9d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3abc7393-c6a9-4443-b05b-c4fe415ea9d9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:18:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 06:18:07,637 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:18:07,639 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:18:08,059 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:18:08,500 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:18:08,522 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:18:08,746 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:18:08,973 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:18:09,454 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:18:09,985 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} builder stdout | 2023-01-17 06:18:10,010 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:10,022 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:10,022 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:10,033 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:18:10,126 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:18:10,371 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:18:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:18:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:40.738568+00:00 (in 29.999505 seconds) buildlogsarchiver stdout | 2023-01-17 06:18:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:40 UTC)" (scheduled at 2023-01-17 06:18:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:18:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 18, 10, 739512), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:18:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:18:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:18:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:18:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:25.646354+00:00 (in 9.999514 seconds) notificationworker stdout | 2023-01-17 06:18:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:25 UTC)" (scheduled at 2023-01-17 06:18:15.646354+00:00) notificationworker stdout | 2023-01-17 06:18:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 15, 647260), True, datetime.datetime(2023, 1, 17, 6, 18, 15, 647260), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:18:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:19.537472+00:00 (in 2.600013 seconds) securityworker stdout | 2023-01-17 06:18:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:46 UTC)" (scheduled at 2023-01-17 06:18:16.936988+00:00) securityworker stdout | 2023-01-17 06:18:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:18:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:18:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:04.936835+00:00 (in 45.590247 seconds) repositorygcworker stdout | 2023-01-17 06:18:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:19 UTC)" (scheduled at 2023-01-17 06:18:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:18:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:18:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:18:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:46.936988+00:00 (in 27.399048 seconds) securityworker stdout | 2023-01-17 06:18:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:49 UTC)" (scheduled at 2023-01-17 06:18:19.537472+00:00) securityworker stdout | 2023-01-17 06:18:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:18:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:18:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:18:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:18:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:18:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:18:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:49 UTC)" executed successfully builder stdout | 2023-01-17 06:18:20,043 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:20,055 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:20,056 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:20,066 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:18:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:51.348605+00:00 (in 29.999479 seconds) gcworker stdout | 2023-01-17 06:18:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:51 UTC)" (scheduled at 2023-01-17 06:18:21.348605+00:00) gcworker stdout | 2023-01-17 06:18:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726701349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:18:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:18:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:18:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:22,060 [231] [DEBUG] [app] Starting request: urn:request:174a4ace-f829-4dfc-8505-ac95c65162aa (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:22,061 [235] [DEBUG] [app] Starting request: urn:request:89e7ef1b-b284-445c-aa58-f6af83ae6814 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,062 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:22,065 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:22,065 [239] [DEBUG] [app] Starting request: urn:request:21a113ed-44e6-4299-832a-3a9afd12c959 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:22,066 [239] [DEBUG] [app] Ending request: urn:request:21a113ed-44e6-4299-832a-3a9afd12c959 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:21a113ed-44e6-4299-832a-3a9afd12c959', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:18:22,066 [240] [DEBUG] [app] Starting request: urn:request:ee0f24ec-a22f-4c39-9773-48974b4299b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:22,066 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:18:22,066 [240] [DEBUG] [app] Ending request: urn:request:ee0f24ec-a22f-4c39-9773-48974b4299b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee0f24ec-a22f-4c39-9773-48974b4299b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:18:22,067 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:18:22,067 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:22,068 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:22,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:22,070 [236] [DEBUG] [app] Starting request: urn:request:ff30b204-247d-4286-b371-1dbd2ac2e34a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:22,071 [236] [DEBUG] [app] Ending request: urn:request:ff30b204-247d-4286-b371-1dbd2ac2e34a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff30b204-247d-4286-b371-1dbd2ac2e34a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:22,071 [233] [DEBUG] [app] Starting request: urn:request:79a23f3a-5e53-438c-a48a-93e3871bc5e5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:18:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:22,071 [233] [DEBUG] [app] Ending request: urn:request:79a23f3a-5e53-438c-a48a-93e3871bc5e5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79a23f3a-5e53-438c-a48a-93e3871bc5e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:22,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:22,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:22,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:22,088 [235] [DEBUG] [app] Ending request: urn:request:89e7ef1b-b284-445c-aa58-f6af83ae6814 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:89e7ef1b-b284-445c-aa58-f6af83ae6814', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:22,088 [231] [DEBUG] [app] Ending request: urn:request:174a4ace-f829-4dfc-8505-ac95c65162aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:174a4ace-f829-4dfc-8505-ac95c65162aa', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:22,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:22,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:18:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:18:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:35.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 06:18:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:35 UTC)" (scheduled at 2023-01-17 06:18:25.646354+00:00) notificationworker stdout | 2023-01-17 06:18:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 25, 647235), True, datetime.datetime(2023, 1, 17, 6, 18, 25, 647235), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:35 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:26,391 [231] [DEBUG] [app] Starting request: urn:request:094c2b06-d733-4260-b87b-a7a5d71bc06f (/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:26,398 [231] [DEBUG] [endpoints.api.discovery] Unable to find method for POST in class gunicorn-web stdout | 2023-01-17 06:18:26,419 [231] [DEBUG] [app] Ending request: urn:request:094c2b06-d733-4260-b87b-a7a5d71bc06f (/) {'endpoint': 'web.index', 'request_id': 'urn:request:094c2b06-d733-4260-b87b-a7a5d71bc06f', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/', 'path': '/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:26,420 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:26 +0000] "GET / HTTP/1.0" 200 84905 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:26 +0000] "GET / HTTP/2.0" 200 13244 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.031 1182 0.031) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/font-awesome.css-a166e3c966c6.css HTTP/2.0" 200 8404 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 293 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/bootstrap.min.css-13f668494557.css HTTP/2.0" 200 24427 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/cssfamilySourceSansPro300400700-30cd793a4b9b.css HTTP/2.0" 200 224 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 72 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/bootstrap-datetimepicker.min.css-4b15a4ede750.css HTTP/2.0" 200 1470 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 72 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/cal-heatmap.css-904c3b9e8a60.css HTTP/2.0" 200 787 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 61 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/ng-tags-input.min.css-7782190f81cd.css HTTP/2.0" 200 1067 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 65 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/jquery.js-3c5c5966471e.js HTTP/2.0" 200 102774 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 67 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/bootstrap.min.js-0e773844b374.js HTTP/2.0" 200 11290 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/angular.min.js-f0f4228b82f7.js HTTP/2.0" 200 63479 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 60 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/angular-route.min.js-590b72b0eb88.js HTTP/2.0" 200 2236 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 64 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/angular-sanitize.min.js-ff589de0ec1b.js HTTP/2.0" 200 3116 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 66 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/angular-animate.min.js-636f7a51b2e1.js HTTP/2.0" 200 10296 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.020 66 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/angular-cookies.min.js-2455ea25d522.js HTTP/2.0" 200 814 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 65 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/momentjs-66a52705d33f.js HTTP/2.0" 200 19464 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 56 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/bootstrap-datepicker.min.js-3e3d8e141d5e.js HTTP/2.0" 200 9310 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 69 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/bootstrap-datetimepicker.min.js-6612bb423b71.js HTTP/2.0" 200 10819 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 72 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/bootbox4.1.0underscorejs1.5.2restangular1.2.0d3js3.3.3-6d47ddfef9c5.js HTTP/2.0" 200 71170 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 88 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/raven.min.js-ecd480647a39.js HTTP/2.0" 200 9492 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 59 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/cal-heatmap.min.js-8ffad728269a.js HTTP/2.0" 200 12991 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 63 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/angular-recaptcha.min.js-23859fca3602.js HTTP/2.0" 200 1781 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 67 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/ng-tags-input.min.js-fa3026ba8409.js HTTP/2.0" 200 6089 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 64 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/ldn/typeahead.bundle.min.js-716151ca4e5d.js HTTP/2.0" 200 16660 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.011 66 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/standalonelib/jquery.overscroll.min.js HTTP/2.0" 200 3479 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.138 63 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:27 +0000] "GET /static/build/main-quay-frontend-e932307d1657af47ffb2.bundle.js HTTP/2.0" 200 719574 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.347 75 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:28 +0000] "GET /static/img/RedHat.svg HTTP/2.0" 200 1233 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 102 -) exportactionlogsworker stdout | 2023-01-17 06:18:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:18:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:35.853396+00:00 (in 6.889226 seconds) exportactionlogsworker stdout | 2023-01-17 06:18:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:28 UTC)" (scheduled at 2023-01-17 06:18:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:18:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:18:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:28 UTC)" executed successfully nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/build/83-quay-frontend.chunk.js HTTP/2.0" 200 1705 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 59 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/build/135-quay-frontend.chunk.js HTTP/2.0" 200 963 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 59 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/build/18-quay-frontend.chunk.js HTTP/2.0" 200 985 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 58 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/build/112-quay-frontend.chunk.js HTTP/2.0" 200 755 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 59 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/build/181-quay-frontend.chunk.js HTTP/2.0" 200 940 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 59 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/build/152-quay-frontend.chunk.js HTTP/2.0" 200 313 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 59 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/img/network-tile.png HTTP/2.0" 200 6289 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 50 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/build/489e5e00e1a9a80f88d55281894a87ff.woff HTTP/2.0" 200 7280 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 150 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/fonts/fontawesome-webfont.woff2?v=4.7.0 HTTP/2.0" 200 77160 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/static/ldn/font-awesome.css-a166e3c966c6.css" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 162 -) gunicorn-web stdout | 2023-01-17 06:18:29,523 [236] [DEBUG] [app] Starting request: urn:request:eae57190-2b75-48eb-b6bc-515da0869d4e (/api/v1/user/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:29,524 [236] [DEBUG] [app] Ending request: urn:request:eae57190-2b75-48eb-b6bc-515da0869d4e (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:eae57190-2b75-48eb-b6bc-515da0869d4e', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /api/v1/user/ HTTP/2.0" 401 311 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.004 160 0.003) gunicorn-web stdout | 2023-01-17 06:18:29,525 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:29 +0000] "GET /api/v1/user/ HTTP/1.0" 401 311 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:18:29,590 [235] [DEBUG] [app] Starting request: urn:request:cd3f1f87-95d0-448a-a781-7ad04f4d6af6 (/api/v1/messages) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:29,591 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."content", "t1"."uuid", "t1"."severity", "t1"."media_type_id", "t2"."id", "t2"."name" FROM "messages" AS "t1" INNER JOIN "mediatype" AS "t2" ON ("t1"."media_type_id" = "t2"."id")', []) gunicorn-web stdout | 2023-01-17 06:18:29,601 [235] [DEBUG] [app] Ending request: urn:request:cd3f1f87-95d0-448a-a781-7ad04f4d6af6 (/api/v1/messages) {'endpoint': 'api.globalusermessages', 'request_id': 'urn:request:cd3f1f87-95d0-448a-a781-7ad04f4d6af6', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/messages', 'path': '/api/v1/messages', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:29,602 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:29,602 [235] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:29 +0000] "GET /api/v1/messages HTTP/1.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /api/v1/messages HTTP/2.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.013 43 0.013) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:29 +0000] "GET /static/img/RH_Logo_Quay_Black_UX-horizontal.svg HTTP/2.0" 200 3210 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 66 -) gunicorn-web stdout | 2023-01-17 06:18:30,007 [236] [DEBUG] [app] Starting request: urn:request:886b5c55-9484-4a2e-bf68-2bb9c5418b98 (/opensearch.xml) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:30,010 [236] [DEBUG] [app] Ending request: urn:request:886b5c55-9484-4a2e-bf68-2bb9c5418b98 (/opensearch.xml) {'endpoint': 'web.opensearch', 'request_id': 'urn:request:886b5c55-9484-4a2e-bf68-2bb9c5418b98', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/opensearch.xml', 'path': '/opensearch.xml', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:30,010 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:30 +0000] "GET /opensearch.xml HTTP/1.0" 200 613 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:30 +0000] "GET /opensearch.xml HTTP/2.0" 200 613 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.004 104 0.004) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:30 +0000] "GET /static/img/quay_favicon.png HTTP/2.0" 200 3558 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 255 -) builder stdout | 2023-01-17 06:18:30,076 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:30,089 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:30,089 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:30,099 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:31,268 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:18:31,342 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:18:31,733 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:18:31,736 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:18:31,738 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:18:31,740 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:18:31,743 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:18:32,274 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:18:32,686 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} builder stdout | 2023-01-17 06:18:32,775 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} namespacegcworker stdout | 2023-01-17 06:18:32,789 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} securityworker stdout | 2023-01-17 06:18:33,214 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:18:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:36.635997+00:00 (in 0.989170 seconds) notificationworker stdout | 2023-01-17 06:18:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:45 UTC)" (scheduled at 2023-01-17 06:18:35.646354+00:00) notificationworker stdout | 2023-01-17 06:18:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 35, 647244), True, datetime.datetime(2023, 1, 17, 6, 18, 35, 647244), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:18:35,714 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:18:35,717 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:18:35,719 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:18:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:18:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:28.963662+00:00 (in 53.109745 seconds) exportactionlogsworker stdout | 2023-01-17 06:18:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:35 UTC)" (scheduled at 2023-01-17 06:18:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:18:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:18:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 35, 854282), True, datetime.datetime(2023, 1, 17, 6, 18, 35, 854282), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:18:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:18:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:18:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:18:36,212 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:18:36,220 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:18:36,224 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:18:36,227 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:18:36,233 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:18:36,235 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:18:36,237 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:18:36,246 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:18:36,249 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:18:36,441 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:18:36,539 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} notificationworker stdout | 2023-01-17 06:18:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:45.646354+00:00 (in 9.009903 seconds) notificationworker stdout | 2023-01-17 06:18:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:23:36 UTC)" (scheduled at 2023-01-17 06:18:36.635997+00:00) notificationworker stdout | 2023-01-17 06:18:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 18, 36, 636802), 'notification/%']) notificationworker stdout | 2023-01-17 06:18:36,647 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 18, 36, 636802), True, datetime.datetime(2023, 1, 17, 6, 18, 36, 636802), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:18:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 18, 36, 636802), True, datetime.datetime(2023, 1, 17, 6, 18, 36, 636802), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 18, 36, 636802), 'notification/%']) notificationworker stdout | 2023-01-17 06:18:36,654 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:36,654 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:23:36 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:37,059 [231] [DEBUG] [app] Starting request: urn:request:b3dd44ae-eb1d-42da-9a99-ab23632d6f6c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:37,059 [233] [DEBUG] [app] Starting request: urn:request:1b541f05-dcaf-461f-aeab-501372e05559 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:37,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:37,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:37,064 [240] [DEBUG] [app] Starting request: urn:request:c0ee3ab9-dd17-47d3-a433-21beb30742e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:37,065 [239] [DEBUG] [app] Starting request: urn:request:6f6ba3ea-659c-4201-b0b4-0aebbd38c7a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:37,065 [240] [DEBUG] [app] Ending request: urn:request:c0ee3ab9-dd17-47d3-a433-21beb30742e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c0ee3ab9-dd17-47d3-a433-21beb30742e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:18:37,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:18:37,065 [239] [DEBUG] [app] Ending request: urn:request:6f6ba3ea-659c-4201-b0b4-0aebbd38c7a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f6ba3ea-659c-4201-b0b4-0aebbd38c7a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:37,065 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:18:37,065 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:18:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:37,068 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:37,069 [235] [DEBUG] [app] Starting request: urn:request:2e882c25-8eb3-49b0-a602-9439faf23865 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:37,070 [236] [DEBUG] [app] Starting request: urn:request:c2350e0d-00cc-45fe-a2e0-db13f57d71d3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:37,070 [235] [DEBUG] [app] Ending request: urn:request:2e882c25-8eb3-49b0-a602-9439faf23865 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e882c25-8eb3-49b0-a602-9439faf23865', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:37,070 [236] [DEBUG] [app] Ending request: urn:request:c2350e0d-00cc-45fe-a2e0-db13f57d71d3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2350e0d-00cc-45fe-a2e0-db13f57d71d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:18:37,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:37,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:37,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:37,086 [233] [DEBUG] [app] Ending request: urn:request:1b541f05-dcaf-461f-aeab-501372e05559 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1b541f05-dcaf-461f-aeab-501372e05559', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:37,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:37,087 [231] [DEBUG] [app] Ending request: urn:request:b3dd44ae-eb1d-42da-9a99-ab23632d6f6c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b3dd44ae-eb1d-42da-9a99-ab23632d6f6c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:37,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:18:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 06:18:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:18:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:37.537407+00:00 (in 0.280331 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:18:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:37 UTC)" (scheduled at 2023-01-17 06:18:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:18:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:18:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 37, 257532), True, datetime.datetime(2023, 1, 17, 6, 18, 37, 257532), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:18:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:18:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:18:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:18:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:18:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:37.256652+00:00 (in 59.718787 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:18:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:37 UTC)" (scheduled at 2023-01-17 06:18:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:18:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:18:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:18:37,661 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:18:37,663 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:18:38,094 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:18:38,533 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:18:38,553 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:18:38,764 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:18:39,008 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:18:39,490 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:18:40,021 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} builder stdout | 2023-01-17 06:18:40,110 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:40,124 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:40,124 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:40,134 [60] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:18:40,160 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:18:40,405 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:18:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:18:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:10.738568+00:00 (in 29.999486 seconds) buildlogsarchiver stdout | 2023-01-17 06:18:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:10 UTC)" (scheduled at 2023-01-17 06:18:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:18:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 18, 40, 739463), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:18:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:18:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:18:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:18:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:18:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:42.941536+00:00 (in 1.792817 seconds) namespacegcworker stdout | 2023-01-17 06:18:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:41 UTC)" (scheduled at 2023-01-17 06:18:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:18:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:18:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 41, 149114), True, datetime.datetime(2023, 1, 17, 6, 18, 41, 149114), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:18:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:18:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:18:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:18:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:18:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:41.148245+00:00 (in 58.206233 seconds) namespacegcworker stdout | 2023-01-17 06:18:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:42 UTC)" (scheduled at 2023-01-17 06:18:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:18:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:18:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:19:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:18:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:55.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 06:18:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:55 UTC)" (scheduled at 2023-01-17 06:18:45.646354+00:00) notificationworker stdout | 2023-01-17 06:18:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 45, 647173), True, datetime.datetime(2023, 1, 17, 6, 18, 45, 647173), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:45,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:18:55 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:46,394 [231] [DEBUG] [app] Starting request: urn:request:4b79ba2b-2d8d-4e34-97a4-7333fc8894a7 (/api/v1/user/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:46,397 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((("t1"."username" = %s) AND ("t1"."organization" = %s)) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,408 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) OR ("t1"."email" = %s)) LIMIT %s OFFSET %s', ['user1', 'bogus1@bogus.com', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,412 [231] [DEBUG] [data.model.user] Email and username are unique! gunicorn-web stdout | 2023-01-17 06:18:46,412 [231] [DEBUG] [peewee] ('INSERT INTO "user" ("uuid", "username", "email", "verified", "organization", "robot", "invoice_email", "invalid_login_attempts", "last_invalid_login", "removed_tag_expiration_s", "enabled", "maximum_queued_builds_count", "creation_date") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "user"."id"', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', 'user1', 'bogus1@bogus.com', False, False, False, False, 0, datetime.datetime(2023, 1, 17, 6, 18, 46, 412326), 1209600, True, None, datetime.datetime(2023, 1, 17, 6, 18, 46, 412327)]) gunicorn-web stdout | 2023-01-17 06:18:46,652 [231] [DEBUG] [peewee] ('UPDATE "user" SET "uuid" = %s, "username" = %s, "password_hash" = %s, "email" = %s, "verified" = %s, "organization" = %s, "robot" = %s, "invoice_email" = %s, "invalid_login_attempts" = %s, "last_invalid_login" = %s, "removed_tag_expiration_s" = %s, "enabled" = %s, "maximum_queued_builds_count" = %s, "creation_date" = %s WHERE ("user"."id" = %s)', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', 'user1', '$2b$12$WTJ2mIAYkn20tj20BY4mZusFqMCP2Fv9lG0rqUIMykjkrQw1r9tLy', 'bogus1@bogus.com', True, False, False, False, 0, datetime.datetime(2023, 1, 17, 6, 18, 46, 412326), 1209600, True, None, datetime.datetime(2023, 1, 17, 6, 18, 46, 412327), 2]) gunicorn-web stdout | 2023-01-17 06:18:46,657 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,661 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,664 [231] [DEBUG] [endpoints.common] Successfully signed in as user user1 with uuid e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,664 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:46,664 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,665 [231] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "team" AS "t2" ON ("t2"."organization_id" = "t1"."id") INNER JOIN "teammember" AS "t3" ON ("t3"."team_id" = "t2"."id") INNER JOIN "user" AS "t4" ON ("t4"."id" = "t3"."user_id") WHERE (("t1"."organization" = %s) AND ("t4"."username" = %s))', [True, 'user1']) gunicorn-web stdout | 2023-01-17 06:18:46,671 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,671 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,674 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:46,674 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:46,675 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:46,675 [231] [DEBUG] [peewee] ('SELECT "t1"."service_ident", "t2"."name", "t1"."metadata_json" FROM "federatedlogin" AS "t1" INNER JOIN "loginservice" AS "t2" ON ("t1"."service_id" = "t2"."id") WHERE (("t2"."name" != %s) AND ("t1"."user_id" = %s))', ['quayrobot', 2]) gunicorn-web stdout | 2023-01-17 06:18:46,680 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."kind_id" FROM "userprompt" AS "t1" INNER JOIN "userpromptkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE ("t1"."user_id" = %s)', [2]) gunicorn-web stdout | 2023-01-17 06:18:46,684 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,687 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:18:46,691 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,695 [231] [DEBUG] [peewee] ('INSERT INTO "repositorysize" ("repository_id", "size_bytes") SELECT "t1"."id", sum("t2"."layers_compressed_size") FROM "manifest" AS "t2" INNER JOIN "repository" AS "t1" ON ("t2"."repository_id" = "t1"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t2"."id") LEFT OUTER JOIN "repositorysize" AS "repositorysize" ON ("repositorysize"."repository_id" = "t1"."id") WHERE (("t1"."namespace_user_id" = %s) AND ("repositorysize"."repository_id" IS %s)) GROUP BY "t1"."id" RETURNING "repositorysize"."id"', [False, None, 1673936326694, 0, 2, None]) gunicorn-web stdout | 2023-01-17 06:18:46,701 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,705 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:18:46,708 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,712 [231] [DEBUG] [peewee] ('SELECT sum("t1"."layers_compressed_size") FROM "manifest" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t1"."id") WHERE ("t2"."namespace_user_id" = %s)', [False, None, 1673936326711, 0, 2]) gunicorn-web stdout | 2023-01-17 06:18:46,715 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,715 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,718 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,719 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,723 [231] [DEBUG] [app] Ending request: urn:request:4b79ba2b-2d8d-4e34-97a4-7333fc8894a7 (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:4b79ba2b-2d8d-4e34-97a4-7333fc8894a7', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': {'username': 'user1', 'email': 'bogus1@bogus.com', 'password': '[FILTERED]', 'repeatPassword': '12345678'}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:46,723 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:46 +0000] "POST /api/v1/user/ HTTP/2.0" 200 639 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.330 1571 0.330) gunicorn-web stdout | 2023-01-17 06:18:46,723 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:46 +0000] "POST /api/v1/user/ HTTP/1.0" 200 639 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" securityworker stdout | 2023-01-17 06:18:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:18:49.537472+00:00 (in 2.599993 seconds) securityworker stdout | 2023-01-17 06:18:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:16 UTC)" (scheduled at 2023-01-17 06:18:46.936988+00:00) securityworker stdout | 2023-01-17 06:18:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:46,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:46,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:46,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:46,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:16 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:46,957 [231] [DEBUG] [app] Starting request: urn:request:5812235f-a90b-4129-8304-e5e41b3294e2 (/api/v1/user/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:46,958 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,958 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,958 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,968 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:46,968 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:46,968 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,968 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,968 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:46,968 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:46,968 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:46,969 [231] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "team" AS "t2" ON ("t2"."organization_id" = "t1"."id") INNER JOIN "teammember" AS "t3" ON ("t3"."team_id" = "t2"."id") INNER JOIN "user" AS "t4" ON ("t4"."id" = "t3"."user_id") WHERE (("t1"."organization" = %s) AND ("t4"."username" = %s))', [True, 'user1']) gunicorn-web stdout | 2023-01-17 06:18:46,973 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:46,973 [231] [DEBUG] [peewee] ('SELECT "t1"."service_ident", "t2"."name", "t1"."metadata_json" FROM "federatedlogin" AS "t1" INNER JOIN "loginservice" AS "t2" ON ("t1"."service_id" = "t2"."id") WHERE (("t2"."name" != %s) AND ("t1"."user_id" = %s))', ['quayrobot', 2]) gunicorn-web stdout | 2023-01-17 06:18:46,977 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."kind_id" FROM "userprompt" AS "t1" INNER JOIN "userpromptkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE ("t1"."user_id" = %s)', [2]) gunicorn-web stdout | 2023-01-17 06:18:46,981 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,984 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:18:46,987 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,991 [231] [DEBUG] [peewee] ('INSERT INTO "repositorysize" ("repository_id", "size_bytes") SELECT "t1"."id", sum("t2"."layers_compressed_size") FROM "manifest" AS "t2" INNER JOIN "repository" AS "t1" ON ("t2"."repository_id" = "t1"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t2"."id") LEFT OUTER JOIN "repositorysize" AS "repositorysize" ON ("repositorysize"."repository_id" = "t1"."id") WHERE (("t1"."namespace_user_id" = %s) AND ("repositorysize"."repository_id" IS %s)) GROUP BY "t1"."id" RETURNING "repositorysize"."id"', [False, None, 1673936326990, 0, 2, None]) gunicorn-web stdout | 2023-01-17 06:18:46,996 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:46,999 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:18:47,003 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:47,006 [231] [DEBUG] [peewee] ('SELECT sum("t1"."layers_compressed_size") FROM "manifest" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t1"."id") WHERE ("t2"."namespace_user_id" = %s)', [False, None, 1673936327006, 0, 2]) gunicorn-web stdout | 2023-01-17 06:18:47,009 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,010 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,010 [231] [DEBUG] [app] Ending request: urn:request:5812235f-a90b-4129-8304-e5e41b3294e2 (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:5812235f-a90b-4129-8304-e5e41b3294e2', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:47,011 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:47,011 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/user/ HTTP/1.0" 200 637 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/user/ HTTP/2.0" 200 637 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.055 437 0.055) gunicorn-web stdout | 2023-01-17 06:18:47,254 [231] [DEBUG] [app] Starting request: urn:request:b82ca4ff-172e-4b66-b607-c95b17374215 (/api/v1/user/notifications) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:47,254 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,254 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,254 [233] [DEBUG] [app] Starting request: urn:request:0ca91f2b-62a0-4b96-950b-5cf1c0526d9d (/api/v1/user/apptoken) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:47,255 [233] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,255 [233] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,255 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:47,256 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [endpoints.api] Checking permission for user user1 gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,265 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,265 [233] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:47,265 [233] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:47,266 [233] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,266 [233] [DEBUG] [endpoints.api] Checking permission for user user1 gunicorn-web stdout | 2023-01-17 06:18:47,266 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,266 [233] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,266 [233] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,266 [233] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,267 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 2, False, 2, 'admin', 6]) gunicorn-web stdout | 2023-01-17 06:18:47,267 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."uuid", "t1"."title", "t1"."token_name", "t1"."token_secret", "t1"."created", "t1"."expiration", "t1"."last_accessed" FROM "appspecificauthtoken" AS "t1" WHERE ((("t1"."user_id" = %s) AND ("t1"."expiration" <= %s)) AND ("t1"."expiration" > %s))', [2, datetime.datetime(2023, 1, 20, 1, 30, 47, 266656), datetime.datetime(2023, 1, 17, 6, 18, 47, 266693)]) gunicorn-web stdout | 2023-01-17 06:18:47,270 [233] [DEBUG] [app] Ending request: urn:request:0ca91f2b-62a0-4b96-950b-5cf1c0526d9d (/api/v1/user/apptoken) {'endpoint': 'api.apptokens', 'request_id': 'urn:request:0ca91f2b-62a0-4b96-950b-5cf1c0526d9d', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/apptoken?expiring=true', 'path': '/api/v1/user/apptoken', 'parameters': {'expiring': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:47,271 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:47,271 [233] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/user/apptoken?expiring=true HTTP/1.0" 200 38 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/user/apptoken?expiring=true HTTP/2.0" 200 38 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 59 0.018) gunicorn-web stdout | 2023-01-17 06:18:47,272 [231] [DEBUG] [app] Ending request: urn:request:b82ca4ff-172e-4b66-b607-c95b17374215 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:b82ca4ff-172e-4b66-b607-c95b17374215', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:47,273 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:47,273 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.020 405 0.020) gunicorn-web stdout | 2023-01-17 06:18:47,605 [236] [DEBUG] [app] Starting request: urn:request:c2efa1c3-0f6e-407f-a46e-548948a1bc3c (/api/v1/repository) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:47,605 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,605 [231] [DEBUG] [app] Starting request: urn:request:b5fa5ee3-180d-4147-a3dd-bcd663d47d8f (/api/v1/repository) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:18:47,605 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,605 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,605 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,606 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:47,606 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:47,616 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:47,616 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:47,616 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,616 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:47,617 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:47,617 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,617 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:47,617 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,617 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,617 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:47,617 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:18:47,618 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['user1', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:47,621 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:18:47,622 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [2, 1, 3]) gunicorn-web stdout | 2023-01-17 06:18:47,624 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:18:47,628 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [2, 1, 3]) gunicorn-web stdout | 2023-01-17 06:18:47,629 [231] [DEBUG] [peewee] ('((SELECT DISTINCT "t1"."name", "t1"."id" AS "rid", "t1"."description", "t2"."username", "t1"."visibility_id", "t1"."kind_id", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") LEFT OUTER JOIN "repositorypermission" AS "t3" ON ("t3"."repository_id" = "t1"."id") WHERE (((("t1"."state" != %s) AND ("t1"."kind_id" = %s)) AND ("t2"."username" = %s)) AND ("t3"."user_id" = %s))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") INNER JOIN "team" AS "t7" ON ("t6"."team_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t8"."team_id" = "t7"."id") WHERE (((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s)) AND ("t8"."user_id" = %s)))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") INNER JOIN "team" AS "t9" ON ("t9"."organization_id" = "t4"."namespace_user_id") INNER JOIN "teammember" AS "t10" ON ("t9"."id" = "t10"."team_id") WHERE ((((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s)) AND ("t9"."role_id" = %s)) AND ("t10"."user_id" = %s))) ORDER BY rid LIMIT %s', [3, 1, 'user1', 2, 3, 1, 'user1', 2, 3, 1, 'user1', 1, 2, 101]) gunicorn-web stdout | 2023-01-17 06:18:47,631 [236] [DEBUG] [app] Ending request: urn:request:c2efa1c3-0f6e-407f-a46e-548948a1bc3c (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:c2efa1c3-0f6e-407f-a46e-548948a1bc3c', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository?last_modified=true&popularity=true&starred=true', 'path': '/api/v1/repository', 'parameters': {'last_modified': 'true', 'popularity': 'true', 'starred': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:47,632 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/repository?last_modified=true&popularity=true&starred=true HTTP/2.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.029 156 0.029) gunicorn-web stdout | 2023-01-17 06:18:47,632 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/repository?last_modified=true&popularity=true&starred=true HTTP/1.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:18:47,636 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [2, 1, 3]) gunicorn-web stdout | 2023-01-17 06:18:47,640 [231] [DEBUG] [app] Ending request: urn:request:b5fa5ee3-180d-4147-a3dd-bcd663d47d8f (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:b5fa5ee3-180d-4147-a3dd-bcd663d47d8f', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository?last_modified=true&namespace=user1&popularity=true"a=true&state=true', 'path': '/api/v1/repository', 'parameters': {'last_modified': 'true', 'namespace': 'user1', 'popularity': 'true', 'quota': 'true', 'state': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:47,640 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:47,641 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/repository?last_modified=true&namespace=user1&popularity=true"a=true&state=true HTTP/1.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:18:47 +0000] "GET /api/v1/repository?last_modified=true&namespace=user1&popularity=true"a=true&state=true HTTP/2.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.037 224 0.036) securityworker stdout | 2023-01-17 06:18:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:18:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:16.936988+00:00 (in 27.399034 seconds) securityworker stdout | 2023-01-17 06:18:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:19 UTC)" (scheduled at 2023-01-17 06:18:49.537472+00:00) securityworker stdout | 2023-01-17 06:18:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:18:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:18:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:18:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:18:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:18:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:18:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:18:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:18:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:18:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:18:49,555 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:18:49,555 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:18:49,555 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:18:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:19 UTC)" executed successfully builder stdout | 2023-01-17 06:18:50,145 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:18:50,157 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:18:50,157 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:18:50,168 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:18:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:21.348605+00:00 (in 29.999542 seconds) gcworker stdout | 2023-01-17 06:18:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:21 UTC)" (scheduled at 2023-01-17 06:18:51.348605+00:00) gcworker stdout | 2023-01-17 06:18:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726731349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:18:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:18:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:18:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:52,059 [235] [DEBUG] [app] Starting request: urn:request:4380bac3-ff1c-4ef4-94f8-a7343e0afb93 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:52,059 [231] [DEBUG] [app] Starting request: urn:request:8253c657-a1b4-40c8-b969-d0795a8adcef (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:18:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:18:52,064 [230] [DEBUG] [app] Starting request: urn:request:22ede0b5-e025-4a98-beac-55211d024453 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:52,065 [240] [DEBUG] [app] Starting request: urn:request:875e0579-8636-4248-a547-085f9ea58094 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:18:52,065 [230] [DEBUG] [app] Ending request: urn:request:22ede0b5-e025-4a98-beac-55211d024453 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:22ede0b5-e025-4a98-beac-55211d024453', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:18:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:18:52,065 [240] [DEBUG] [app] Ending request: urn:request:875e0579-8636-4248-a547-085f9ea58094 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:875e0579-8636-4248-a547-085f9ea58094', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:52,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:18:52,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:18:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:18:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:18:52,070 [236] [DEBUG] [app] Starting request: urn:request:8d3d92d8-3f3b-4fc8-ba00-db378fe3fbb3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:52,070 [235] [DEBUG] [app] Starting request: urn:request:aad667d0-5ab8-43f0-90d3-ab4389ce6fd7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:18:52,070 [236] [DEBUG] [app] Ending request: urn:request:8d3d92d8-3f3b-4fc8-ba00-db378fe3fbb3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d3d92d8-3f3b-4fc8-ba00-db378fe3fbb3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:52,070 [235] [DEBUG] [app] Ending request: urn:request:aad667d0-5ab8-43f0-90d3-ab4389ce6fd7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aad667d0-5ab8-43f0-90d3-ab4389ce6fd7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:18:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:18:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:18:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:18:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:18:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:18:52,087 [235] [DEBUG] [app] Ending request: urn:request:4380bac3-ff1c-4ef4-94f8-a7343e0afb93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4380bac3-ff1c-4ef4-94f8-a7343e0afb93', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:52,088 [231] [DEBUG] [app] Ending request: urn:request:8253c657-a1b4-40c8-b969-d0795a8adcef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8253c657-a1b4-40c8-b969-d0795a8adcef', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:18:52,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:18:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:18:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 06:18:55,382 [231] [DEBUG] [app] Starting request: urn:request:a0a272f1-1432-45dc-b657-97c3833ea3f1 (/api/v1/organization/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:55,382 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,383 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,383 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,394 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:55,394 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:55,394 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,394 [231] [DEBUG] [endpoints.api] Checking permission for user user1 gunicorn-web stdout | 2023-01-17 06:18:55,395 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,395 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,395 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,395 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,396 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,400 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,404 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) OR ("t1"."email" = %s)) LIMIT %s OFFSET %s', ['build_test', '1df20627-a737-4aaa-9b6b-99345f8ca6ab', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,406 [231] [DEBUG] [data.model.user] Email and username are unique! gunicorn-web stdout | 2023-01-17 06:18:55,407 [231] [DEBUG] [peewee] ('INSERT INTO "user" ("uuid", "username", "email", "verified", "organization", "robot", "invoice_email", "invalid_login_attempts", "last_invalid_login", "removed_tag_expiration_s", "enabled", "maximum_queued_builds_count", "creation_date") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "user"."id"', ['f4e76b1f-0f02-4768-bffb-481123f494a2', 'build_test', '1df20627-a737-4aaa-9b6b-99345f8ca6ab', False, False, False, False, 0, datetime.datetime(2023, 1, 17, 6, 18, 55, 406787), 1209600, True, None, datetime.datetime(2023, 1, 17, 6, 18, 55, 406788)]) gunicorn-web stdout | 2023-01-17 06:18:55,408 [231] [DEBUG] [peewee] ('UPDATE "user" SET "uuid" = %s, "username" = %s, "email" = %s, "verified" = %s, "organization" = %s, "robot" = %s, "invoice_email" = %s, "invalid_login_attempts" = %s, "last_invalid_login" = %s, "removed_tag_expiration_s" = %s, "enabled" = %s, "maximum_queued_builds_count" = %s, "creation_date" = %s WHERE ("user"."id" = %s)', ['f4e76b1f-0f02-4768-bffb-481123f494a2', 'build_test', '1df20627-a737-4aaa-9b6b-99345f8ca6ab', False, True, False, False, 0, datetime.datetime(2023, 1, 17, 6, 18, 55, 406787), 1209600, True, None, datetime.datetime(2023, 1, 17, 6, 18, 55, 406788), 3]) gunicorn-web stdout | 2023-01-17 06:18:55,410 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['admin', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,411 [231] [DEBUG] [peewee] ('INSERT INTO "team" ("name", "organization_id", "role_id", "description") VALUES (%s, %s, %s, %s) RETURNING "team"."id"', ['owners', 3, 1, '']) gunicorn-web stdout | 2023-01-17 06:18:55,414 [231] [DEBUG] [peewee] ('INSERT INTO "teammember" ("user_id", "team_id") VALUES (%s, %s) RETURNING "teammember"."id"', [2, 1]) gunicorn-web stdout | 2023-01-17 06:18:55,419 [231] [DEBUG] [app] Ending request: urn:request:a0a272f1-1432-45dc-b657-97c3833ea3f1 (/api/v1/organization/) {'endpoint': 'api.organizationlist', 'request_id': 'urn:request:a0a272f1-1432-45dc-b657-97c3833ea3f1', 'remote_addr': '10.131.0.6', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/organization/', 'path': '/api/v1/organization/', 'parameters': {}, 'json_body': {'name': 'build_test'}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:55,419 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:55 +0000] "POST /api/v1/organization/ HTTP/2.0" 201 10 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organizations/new/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.038 1725 0.038) gunicorn-web stdout | 2023-01-17 06:18:55,420 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:55 +0000] "POST /api/v1/organization/ HTTP/1.0" 201 10 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organizations/new/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" notificationworker stdout | 2023-01-17 06:18:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:18:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:01.636362+00:00 (in 5.989572 seconds) notificationworker stdout | 2023-01-17 06:18:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:05 UTC)" (scheduled at 2023-01-17 06:18:55.646354+00:00) notificationworker stdout | 2023-01-17 06:18:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:18:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 18, 55, 647172), True, datetime.datetime(2023, 1, 17, 6, 18, 55, 647172), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:18:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:18:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:18:55,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:05 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:18:55,696 [236] [DEBUG] [app] Starting request: urn:request:4ffdad9a-d46c-420d-a4c6-d0319002f106 (/api/v1/user/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:55,696 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,696 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,697 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,706 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:55,706 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:55,706 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,707 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,707 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,707 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,707 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,708 [236] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "team" AS "t2" ON ("t2"."organization_id" = "t1"."id") INNER JOIN "teammember" AS "t3" ON ("t3"."team_id" = "t2"."id") INNER JOIN "user" AS "t4" ON ("t4"."id" = "t3"."user_id") WHERE (("t1"."organization" = %s) AND ("t4"."username" = %s))', [True, 'user1']) gunicorn-web stdout | 2023-01-17 06:18:55,712 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,712 [236] [DEBUG] [peewee] ('SELECT "t1"."service_ident", "t2"."name", "t1"."metadata_json" FROM "federatedlogin" AS "t1" INNER JOIN "loginservice" AS "t2" ON ("t1"."service_id" = "t2"."id") WHERE (("t2"."name" != %s) AND ("t1"."user_id" = %s))', ['quayrobot', 2]) gunicorn-web stdout | 2023-01-17 06:18:55,716 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."kind_id" FROM "userprompt" AS "t1" INNER JOIN "userpromptkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE ("t1"."user_id" = %s)', [2]) gunicorn-web stdout | 2023-01-17 06:18:55,720 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,723 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:18:55,726 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,731 [236] [DEBUG] [peewee] ('INSERT INTO "repositorysize" ("repository_id", "size_bytes") SELECT "t1"."id", sum("t2"."layers_compressed_size") FROM "manifest" AS "t2" INNER JOIN "repository" AS "t1" ON ("t2"."repository_id" = "t1"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t2"."id") LEFT OUTER JOIN "repositorysize" AS "repositorysize" ON ("repositorysize"."repository_id" = "t1"."id") WHERE (("t1"."namespace_user_id" = %s) AND ("repositorysize"."repository_id" IS %s)) GROUP BY "t1"."id" RETURNING "repositorysize"."id"', [False, None, 1673936335730, 0, 2, None]) gunicorn-web stdout | 2023-01-17 06:18:55,735 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,738 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:18:55,742 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,744 [235] [DEBUG] [app] Starting request: urn:request:c4818cdc-f4ff-4285-b9f7-beb395fee541 (/api/v1/organization/build_test) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:55,744 [235] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,744 [235] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,745 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,746 [236] [DEBUG] [peewee] ('SELECT sum("t1"."layers_compressed_size") FROM "manifest" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t1"."id") WHERE ("t2"."namespace_user_id" = %s)', [False, None, 1673936335745, 0, 2]) gunicorn-web stdout | 2023-01-17 06:18:55,749 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,749 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,750 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:18:55,754 [236] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,754 [236] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,754 [236] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,754 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,754 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,755 [236] [DEBUG] [app] Ending request: urn:request:4ffdad9a-d46c-420d-a4c6-d0319002f106 (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:4ffdad9a-d46c-420d-a4c6-d0319002f106', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:55,755 [235] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:55,755 [235] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:55,755 [235] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,755 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:55,756 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:55 +0000] "GET /api/v1/user/ HTTP/2.0" 200 898 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organizations/new/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.061 410 0.061) gunicorn-web stdout | 2023-01-17 06:18:55,756 [236] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:55 +0000] "GET /api/v1/user/ HTTP/1.0" 200 898 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organizations/new/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:18:55,759 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,759 [235] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,759 [235] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,759 [235] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,760 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:18:55,764 [235] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,764 [235] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,764 [235] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:18:55,765 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t2" ON ("t1"."role_id" = "t2"."id") WHERE ("t1"."organization_id" = %s)', [3]) gunicorn-web stdout | 2023-01-17 06:18:55,768 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:18:55,771 [235] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "repositorypermission" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2023-01-17 06:18:55,775 [235] [DEBUG] [peewee] ('SELECT "t1"."team_id", Count("t1"."id") FROM "teammember" AS "t1" WHERE ("t1"."team_id" IN (%s)) GROUP BY "t1"."team_id"', [1]) gunicorn-web stdout | 2023-01-17 06:18:55,777 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,778 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,778 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:55,778 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2023-01-17 06:18:55,781 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,786 [235] [DEBUG] [peewee] ('INSERT INTO "repositorysize" ("repository_id", "size_bytes") SELECT "t1"."id", sum("t2"."layers_compressed_size") FROM "manifest" AS "t2" INNER JOIN "repository" AS "t1" ON ("t2"."repository_id" = "t1"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t2"."id") LEFT OUTER JOIN "repositorysize" AS "repositorysize" ON ("repositorysize"."repository_id" = "t1"."id") WHERE (("t1"."namespace_user_id" = %s) AND ("repositorysize"."repository_id" IS %s)) GROUP BY "t1"."id" RETURNING "repositorysize"."id"', [False, None, 1673936335785, 0, 3, None]) gunicorn-web stdout | 2023-01-17 06:18:55,790 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,794 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2023-01-17 06:18:55,797 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:55,801 [235] [DEBUG] [peewee] ('SELECT sum("t1"."layers_compressed_size") FROM "manifest" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t1"."id") WHERE ("t2"."namespace_user_id" = %s)', [False, None, 1673936335800, 0, 3]) gunicorn-web stdout | 2023-01-17 06:18:55,804 [235] [DEBUG] [app] Ending request: urn:request:c4818cdc-f4ff-4285-b9f7-beb395fee541 (/api/v1/organization/build_test) {'endpoint': 'api.organization', 'request_id': 'urn:request:c4818cdc-f4ff-4285-b9f7-beb395fee541', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/organization/build_test', 'path': '/api/v1/organization/build_test', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:55,805 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:55,805 [235] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:55 +0000] "GET /api/v1/organization/build_test HTTP/1.0" 200 762 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:55 +0000] "GET /api/v1/organization/build_test HTTP/2.0" 200 762 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.062 139 0.062) gunicorn-web stdout | 2023-01-17 06:18:56,033 [231] [DEBUG] [app] Starting request: urn:request:e106690a-fe8b-4ed2-b8fa-375258a907e9 (/api/v1/user/apptoken) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:56,033 [236] [DEBUG] [app] Starting request: urn:request:8fefc8ef-40d7-465e-8a9a-f64ffd5ff270 (/api/v1/user/notifications) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:56,034 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,034 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,034 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,034 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,034 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,034 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [endpoints.api] Checking permission for user user1 gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,045 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [endpoints.api] Checking permission for user user1 gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,045 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,046 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."uuid", "t1"."title", "t1"."token_name", "t1"."token_secret", "t1"."created", "t1"."expiration", "t1"."last_accessed" FROM "appspecificauthtoken" AS "t1" WHERE ((("t1"."user_id" = %s) AND ("t1"."expiration" <= %s)) AND ("t1"."expiration" > %s))', [2, datetime.datetime(2023, 1, 20, 1, 30, 56, 45725), datetime.datetime(2023, 1, 17, 6, 18, 56, 45761)]) gunicorn-web stdout | 2023-01-17 06:18:56,047 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 2, False, 2, 'admin', 6]) gunicorn-web stdout | 2023-01-17 06:18:56,049 [231] [DEBUG] [app] Ending request: urn:request:e106690a-fe8b-4ed2-b8fa-375258a907e9 (/api/v1/user/apptoken) {'endpoint': 'api.apptokens', 'request_id': 'urn:request:e106690a-fe8b-4ed2-b8fa-375258a907e9', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/apptoken?expiring=true', 'path': '/api/v1/user/apptoken', 'parameters': {'expiring': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:56,050 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:56,050 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/user/apptoken?expiring=true HTTP/1.0" 200 38 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/user/apptoken?expiring=true HTTP/2.0" 200 38 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 60 0.018) gunicorn-web stdout | 2023-01-17 06:18:56,053 [236] [DEBUG] [app] Ending request: urn:request:8fefc8ef-40d7-465e-8a9a-f64ffd5ff270 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:8fefc8ef-40d7-465e-8a9a-f64ffd5ff270', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:56,053 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:56,053 [236] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.021 52 0.021) gunicorn-web stdout | 2023-01-17 06:18:56,297 [236] [DEBUG] [app] Starting request: urn:request:5c0896cf-fd13-4d07-bbcf-7bc165842066 (/api/v1/organization/build_test/quota) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:56,297 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,297 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,297 [231] [DEBUG] [app] Starting request: urn:request:792f981c-d309-4a39-8449-3a9c7802612a (/api/v1/organization/build_test/proxycache) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:56,297 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,297 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,298 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,298 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,308 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:56,308 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:56,308 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:56,308 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:56,308 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,308 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,308 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,308 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,308 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,308 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,308 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,308 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,308 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,308 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,309 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:18:56,310 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:18:56,314 [231] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,314 [236] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,314 [231] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,314 [236] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,314 [231] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,314 [236] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,314 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."organization_id", "t1"."creation_date", "t1"."upstream_registry", "t1"."upstream_registry_username", "t1"."upstream_registry_password", "t1"."expiration_s", "t1"."insecure" FROM "proxycacheconfig" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE (("t2"."username" = %s) AND ("t2"."organization" = %s)) LIMIT %s OFFSET %s', ['build_test', True, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,314 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,318 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test']) gunicorn-web stdout | 2023-01-17 06:18:56,318 [231] [DEBUG] [app] Ending request: urn:request:792f981c-d309-4a39-8449-3a9c7802612a (/api/v1/organization/build_test/proxycache) {'endpoint': 'api.organizationproxycacheconfig', 'request_id': 'urn:request:792f981c-d309-4a39-8449-3a9c7802612a', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/proxycache', 'path': '/api/v1/organization/build_test/proxycache', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:56,319 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:56,319 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/organization/build_test/proxycache HTTP/1.0" 200 62 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/organization/build_test/proxycache HTTP/2.0" 200 62 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.023 377 0.023) gunicorn-web stdout | 2023-01-17 06:18:56,321 [236] [DEBUG] [app] Ending request: urn:request:5c0896cf-fd13-4d07-bbcf-7bc165842066 (/api/v1/organization/build_test/quota) {'endpoint': 'api.organizationquotalist', 'request_id': 'urn:request:5c0896cf-fd13-4d07-bbcf-7bc165842066', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/quota', 'path': '/api/v1/organization/build_test/quota', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:56,322 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:56,322 [236] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/organization/build_test/quota HTTP/1.0" 200 3 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/organization/build_test/quota HTTP/2.0" 200 3 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.026 407 0.026) gunicorn-web stdout | 2023-01-17 06:18:56,515 [231] [DEBUG] [app] Starting request: urn:request:a4dd6b8c-7a02-452e-94c8-b43a62ec79f1 (/api/v1/organization/build_test/prototypes) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:56,515 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,515 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,515 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,525 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:56,525 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:56,525 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,526 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,526 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,526 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,526 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,527 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:18:56,531 [231] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,531 [231] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,531 [231] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,531 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."organization" = %s) AND ("t1"."username" = %s)) LIMIT %s OFFSET %s', [True, 'build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,535 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."org_id", "t1"."uuid", "t1"."activating_user_id", "t1"."delegate_user_id", "t1"."delegate_team_id", "t1"."role_id" FROM "permissionprototype" AS "t1" LEFT OUTER JOIN "user" AS "t2" ON ("t2"."id" = "t1"."activating_user_id") LEFT OUTER JOIN "user" AS "t3" ON ("t3"."id" = "t1"."delegate_user_id") LEFT OUTER JOIN "team" AS "t4" ON ("t4"."id" = "t1"."delegate_team_id") LEFT OUTER JOIN "role" AS "t5" ON ("t5"."id" = "t1"."role_id") WHERE ("t1"."org_id" = %s)', [3]) gunicorn-web stdout | 2023-01-17 06:18:56,540 [231] [DEBUG] [app] Ending request: urn:request:a4dd6b8c-7a02-452e-94c8-b43a62ec79f1 (/api/v1/organization/build_test/prototypes) {'endpoint': 'api.permissionprototypelist', 'request_id': 'urn:request:a4dd6b8c-7a02-452e-94c8-b43a62ec79f1', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/organization/build_test/prototypes', 'path': '/api/v1/organization/build_test/prototypes', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:56,541 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:18:56,541 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/organization/build_test/prototypes HTTP/1.0" 200 19 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/organization/build_test/prototypes HTTP/2.0" 200 19 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.027 190 0.028) nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:56 +0000] "GET /static/img/creditcards/credit.png HTTP/2.0" 200 1776 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 825 -) gunicorn-web stdout | 2023-01-17 06:18:56,598 [231] [DEBUG] [app] Starting request: urn:request:bfa7746a-7673-45cd-9a2d-1fed3bef7211 (/api/v1/repository) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:18:56,598 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,598 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,599 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,609 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:18:56,609 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:18:56,609 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,609 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:18:56,609 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,609 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,609 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:18:56,610 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['user1', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,613 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['public', 1, 0]) gunicorn-web stdout | 2023-01-17 06:18:56,619 [231] [DEBUG] [peewee] ('(((SELECT DISTINCT "t1"."name", "t1"."id" AS "rid", "t1"."description", "t2"."username", "t1"."visibility_id", "t1"."kind_id", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") LEFT OUTER JOIN "repositorypermission" AS "t3" ON ("t3"."repository_id" = "t1"."id") WHERE (((("t1"."state" != %s) AND ("t1"."kind_id" = %s)) AND ("t2"."username" = %s)) AND ("t1"."visibility_id" = %s))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") WHERE (((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s)) AND ("t6"."user_id" = %s)))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") INNER JOIN "team" AS "t7" ON ("t6"."team_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t8"."team_id" = "t7"."id") WHERE (((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s)) AND ("t8"."user_id" = %s)))) UNION (SELECT DISTINCT "t4"."name", "t4"."id" AS "rid", "t4"."description", "t5"."username", "t4"."visibility_id", "t4"."kind_id", "t4"."state" FROM "repository" AS "t4" INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "repositorypermission" AS "t6" ON ("t6"."repository_id" = "t4"."id") INNER JOIN "team" AS "t9" ON ("t9"."organization_id" = "t4"."namespace_user_id") INNER JOIN "teammember" AS "t10" ON ("t9"."id" = "t10"."team_id") WHERE ((((("t4"."state" != %s) AND ("t4"."kind_id" = %s)) AND ("t5"."username" = %s)) AND ("t9"."role_id" = %s)) AND ("t10"."user_id" = %s))) ORDER BY rid LIMIT %s', [3, 1, 'build_test', 1, 3, 1, 'build_test', 2, 3, 1, 'build_test', 2, 3, 1, 'build_test', 1, 2, 101]) gunicorn-web stdout | 2023-01-17 06:18:56,625 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name", "t1"."id" AS "rid" FROM "repository" AS "t1" INNER JOIN "star" AS "t4" ON ("t4"."repository_id" = "t1"."id") INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE ((("t4"."user_id" = %s) AND ("t1"."kind_id" = %s)) AND ("t1"."state" != %s))', [2, 1, 3]) gunicorn-web stdout | 2023-01-17 06:18:56,629 [231] [DEBUG] [app] Ending request: urn:request:bfa7746a-7673-45cd-9a2d-1fed3bef7211 (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:bfa7746a-7673-45cd-9a2d-1fed3bef7211', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository?last_modified=true&namespace=build_test&popularity=true&public=true"a=true', 'path': '/api/v1/repository', 'parameters': {'last_modified': 'true', 'namespace': 'build_test', 'popularity': 'true', 'public': 'true', 'quota': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:18:56,630 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&popularity=true&public=true"a=true HTTP/2.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.033 105 0.032) gunicorn-web stdout | 2023-01-17 06:18:56,630 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:18:56 +0000] "GET /api/v1/repository?last_modified=true&namespace=build_test&popularity=true&public=true"a=true HTTP/1.0" 200 21 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/organization/build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:18:57 +0000] "GET /static/fonts/glyphicons-halflings-regular.woff2 HTTP/2.0" 200 18028 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/static/ldn/bootstrap.min.css-13f668494557.css" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 263 -) builder stdout | 2023-01-17 06:19:00,178 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:00,191 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:00,191 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:00,202 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:19:01,303 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:19:01,371 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:19:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:05.646354+00:00 (in 4.009645 seconds) notificationworker stdout | 2023-01-17 06:19:01,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:01 UTC)" (scheduled at 2023-01-17 06:19:01.636362+00:00) notificationworker stdout | 2023-01-17 06:19:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:19:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:01,742 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:19:01,746 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:19:01,748 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:19:01,751 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:19:01,753 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:19:02,300 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:19:02,722 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:19:02,802 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:19:02,810 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:19:03,250 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:19:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:19:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:19.346120+00:00 (in 14.408843 seconds) repositorygcworker stdout | 2023-01-17 06:19:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:04 UTC)" (scheduled at 2023-01-17 06:19:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:19:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:19:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 4, 937781), True, datetime.datetime(2023, 1, 17, 6, 19, 4, 937781), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:19:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:19:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:19:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:19:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:15.646354+00:00 (in 9.999577 seconds) notificationworker stdout | 2023-01-17 06:19:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:15 UTC)" (scheduled at 2023-01-17 06:19:05.646354+00:00) notificationworker stdout | 2023-01-17 06:19:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 5, 647201), True, datetime.datetime(2023, 1, 17, 6, 19, 5, 647201), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:15 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:05,704 [236] [DEBUG] [app] Starting request: urn:request:86405ef3-6ce6-4277-83bc-81a81ccc8a72 (/api/v1/repository) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:05,704 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:05,704 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:05,705 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,714 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:05,714 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:05,714 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:05,715 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:05,716 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:05,716 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:05,716 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:05,717 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:19:05,721 [236] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-secscan stdout | 2023-01-17 06:19:05,721 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-web stdout | 2023-01-17 06:19:05,721 [236] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:19:05,721 [236] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:19:05,722 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-secscan stdout | 2023-01-17 06:19:05,724 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:19:05,726 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-web stdout | 2023-01-17 06:19:05,727 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,731 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,734 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:19:05,735 [236] [DEBUG] [peewee] ('INSERT INTO "repository" ("namespace_user_id", "name", "visibility_id", "description", "badge_token", "kind_id", "trust_enabled", "state") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "repository"."id"', [3, 'bitbucket', 2, '', 'b4e9a60a-75cd-472a-9193-30aa274a686e', 1, False, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,739 [236] [DEBUG] [peewee] ('INSERT INTO "repositoryactioncount" ("repository_id", "count", "date") VALUES (%s, %s, %s) RETURNING "repositoryactioncount"."id"', [1, 0, datetime.date(2023, 1, 16)]) gunicorn-web stdout | 2023-01-17 06:19:05,741 [236] [DEBUG] [peewee] ('INSERT INTO "repositorysearchscore" ("repository_id", "score") VALUES (%s, %s) RETURNING "repositorysearchscore"."id"', [1, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,743 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "role" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['admin', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,745 [236] [DEBUG] [peewee] ('INSERT INTO "repositorypermission" ("user_id", "repository_id", "role_id") VALUES (%s, %s, %s) RETURNING "repositorypermission"."id"', [2, 1, 1]) gunicorn-web stdout | 2023-01-17 06:19:05,750 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."org_id", "t1"."uuid", "t1"."activating_user_id", "t1"."delegate_user_id", "t1"."delegate_team_id", "t1"."role_id" FROM "permissionprototype" AS "t1" WHERE ((("t1"."org_id" = %s) AND (("t1"."activating_user_id" = %s) OR ("t1"."activating_user_id" IS %s))) AND ("t1"."delegate_user_id" IS %s))', [3, 2, None, None]) gunicorn-web stdout | 2023-01-17 06:19:05,754 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."org_id", "t1"."uuid", "t1"."activating_user_id", "t1"."delegate_user_id", "t1"."delegate_team_id", "t1"."role_id" FROM "permissionprototype" AS "t1" WHERE ((("t1"."org_id" = %s) AND (("t1"."activating_user_id" = %s) OR ("t1"."activating_user_id" IS %s))) AND ("t1"."delegate_team_id" IS %s))', [3, 2, None, None]) gunicorn-web stdout | 2023-01-17 06:19:05,757 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,761 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:05,764 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "logentrykind" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:19:05,769 [236] [DEBUG] [peewee] ('INSERT INTO "logentry3" ("kind_id", "account_id", "performer_id", "repository_id", "datetime", "ip", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "logentry3"."id"', [14, 3, 2, 1, datetime.datetime(2023, 1, 17, 6, 19, 5, 761220), '10.128.2.7', '{"repo": "bitbucket", "namespace": "build_test"}']) gunicorn-web stdout | 2023-01-17 06:19:05,776 [236] [DEBUG] [app] Ending request: urn:request:86405ef3-6ce6-4277-83bc-81a81ccc8a72 (/api/v1/repository) {'endpoint': 'api.repositorylist', 'request_id': 'urn:request:86405ef3-6ce6-4277-83bc-81a81ccc8a72', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository', 'path': '/api/v1/repository', 'parameters': {}, 'json_body': {'namespace': 'build_test', 'repository': 'bitbucket', 'visibility': 'private', 'description': '', 'repo_kind': 'image'}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:05,777 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:05,777 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:05 +0000] "POST /api/v1/repository HTTP/1.0" 201 66 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/new/?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:05 +0000] "POST /api/v1/repository HTTP/2.0" 201 66 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/new/?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.074 1869 0.075) gunicorn-web stdout | 2023-01-17 06:19:06,165 [236] [DEBUG] [app] Starting request: urn:request:bcfd80a1-69f2-4b03-bef3-c224882eb2a0 (/api/v1/repository/build_test/bitbucket/tag/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:06,165 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,165 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,166 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,169 [233] [DEBUG] [app] Starting request: urn:request:2e375877-358d-409f-8677-feee900c32e4 (/api/v1/repository/build_test/bitbucket) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:06,169 [233] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,169 [233] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,170 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,176 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,178 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:06,179 [233] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:06,180 [233] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:06,180 [233] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,180 [233] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:06,180 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,180 [233] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,180 [233] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,180 [233] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,182 [233] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:06,185 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,186 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,189 [233] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:06,189 [233] [DEBUG] [endpoints.api.repository] Get repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:06,190 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,191 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,194 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."repository_id", "t1"."created" FROM "star" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."user_id" = %s)) LIMIT %s OFFSET %s', [1, 2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,194 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['public', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,197 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['public', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,198 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id", "t2"."id", "t2"."digest", "t2"."media_type_id", "t2"."layers_compressed_size", "t2"."config_media_type" FROM "tag" AS "t1" INNER JOIN "manifest" AS "t2" ON ("t1"."manifest_id" = "t2"."id") WHERE ((("t1"."repository_id" = %s) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" > %s))) AND ("t1"."hidden" = %s)) ORDER BY "t1"."lifetime_start_ms" DESC, "t1"."name" LIMIT %s OFFSET %s', [1, None, 1673936346197, False, 101, 0]) gunicorn-web stdout | 2023-01-17 06:19:06,200 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:19:06,202 [236] [DEBUG] [app] Ending request: urn:request:bcfd80a1-69f2-4b03-bef3-c224882eb2a0 (/api/v1/repository/build_test/bitbucket/tag/) {'endpoint': 'api.listrepositorytags', 'request_id': 'urn:request:bcfd80a1-69f2-4b03-bef3-c224882eb2a0', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/tag/?limit=100&page=1&onlyActiveTags=true', 'path': '/api/v1/repository/build_test/bitbucket/tag/', 'parameters': {'limit': '100', 'page': '1', 'onlyActiveTags': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:06,202 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:06,203 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:06 +0000] "GET /api/v1/repository/build_test/bitbucket/tag/?limit=100&page=1&onlyActiveTags=true HTTP/1.0" 200 49 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:06 +0000] "GET /api/v1/repository/build_test/bitbucket/tag/?limit=100&page=1&onlyActiveTags=true HTTP/2.0" 200 49 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.039 544 0.039) gunicorn-web stdout | 2023-01-17 06:19:06,204 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."count", "t1"."date" FROM "repositoryactioncount" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."date" >= %s))', [1, datetime.date(2022, 10, 17)]) gunicorn-web stdout | 2023-01-17 06:19:06,207 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,208 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:06,208 [233] [DEBUG] [app] Ending request: urn:request:2e375877-358d-409f-8677-feee900c32e4 (/api/v1/repository/build_test/bitbucket) {'endpoint': 'api.repository', 'request_id': 'urn:request:2e375877-358d-409f-8677-feee900c32e4', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket?includeStats=true&includeTags=false', 'path': '/api/v1/repository/build_test/bitbucket', 'parameters': {'includeStats': 'true', 'includeTags': 'false'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:06,209 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:06,209 [233] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:06 +0000] "GET /api/v1/repository/build_test/bitbucket?includeStats=true&includeTags=false HTTP/1.0" 200 3629 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:06 +0000] "GET /api/v1/repository/build_test/bitbucket?includeStats=true&includeTags=false HTTP/2.0" 200 3629 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.041 150 0.042) gunicorn-registry stdout | 2023-01-17 06:19:06,221 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:19:06,228 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:19:06,231 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:19:06,238 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:19:06,242 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:19:06,245 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:19:06,248 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:19:06,254 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:19:06,257 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:19:06,454 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:19:06,566 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:19:07,060 [236] [DEBUG] [app] Starting request: urn:request:3af5269e-626e-4d29-800a-55bf0377b57c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:07,060 [233] [DEBUG] [app] Starting request: urn:request:e7fc4417-79fd-4a7b-8b77-fac9e31e08e1 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:07,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:07,065 [230] [DEBUG] [app] Starting request: urn:request:145b75db-90b9-4df3-95f5-c82a1ded6562 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:07,065 [242] [DEBUG] [app] Starting request: urn:request:f25df0f4-ec60-4f84-85b0-4747b19ecd70 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:07,065 [230] [DEBUG] [app] Ending request: urn:request:145b75db-90b9-4df3-95f5-c82a1ded6562 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:145b75db-90b9-4df3-95f5-c82a1ded6562', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:19:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:19:07,066 [242] [DEBUG] [app] Ending request: urn:request:f25df0f4-ec60-4f84-85b0-4747b19ecd70 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f25df0f4-ec60-4f84-85b0-4747b19ecd70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:19:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:19:07,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:07,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:07,070 [231] [DEBUG] [app] Starting request: urn:request:b2ba9107-6162-4c00-bcc6-0f98115bb3e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:07,070 [236] [DEBUG] [app] Starting request: urn:request:c504e5dd-f175-4ed1-aaee-f941398a5663 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:07,070 [231] [DEBUG] [app] Ending request: urn:request:b2ba9107-6162-4c00-bcc6-0f98115bb3e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2ba9107-6162-4c00-bcc6-0f98115bb3e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:07,070 [236] [DEBUG] [app] Ending request: urn:request:c504e5dd-f175-4ed1-aaee-f941398a5663 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c504e5dd-f175-4ed1-aaee-f941398a5663', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:07,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:07,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:07,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:07,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:07,088 [236] [DEBUG] [app] Ending request: urn:request:3af5269e-626e-4d29-800a-55bf0377b57c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3af5269e-626e-4d29-800a-55bf0377b57c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:07,088 [233] [DEBUG] [app] Ending request: urn:request:e7fc4417-79fd-4a7b-8b77-fac9e31e08e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e7fc4417-79fd-4a7b-8b77-fac9e31e08e1', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:07,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:07,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:19:07,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 06:19:07,142 [233] [DEBUG] [app] Starting request: urn:request:10b2e987-49c0-465c-b62f-5d708a47cba6 (/repository/build_test/bitbucket/status) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:07,143 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:07,153 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:07,157 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t3"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 'public', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:07,162 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s OFFSET %s', ['bitbucket', 'build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:07,170 [233] [DEBUG] [app] Ending request: urn:request:10b2e987-49c0-465c-b62f-5d708a47cba6 (/repository/build_test/bitbucket/status) {'endpoint': 'web.build_status_badge', 'request_id': 'urn:request:10b2e987-49c0-465c-b62f-5d708a47cba6', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/status?token=b4e9a60a-75cd-472a-9193-30aa274a686e', 'path': '/repository/build_test/bitbucket/status', 'parameters': {'token': 'b4e9a60a-75cd-472a-9193-30aa274a686e'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:07,171 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:07,171 [233] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:07 +0000] "GET /repository/build_test/bitbucket/status?token=b4e9a60a-75cd-472a-9193-30aa274a686e HTTP/1.0" 200 740 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /repository/build_test/bitbucket/status?token=b4e9a60a-75cd-472a-9193-30aa274a686e HTTP/2.0" 200 387 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.030 501 0.031) gunicorn-web stdout | 2023-01-17 06:19:07,354 [236] [DEBUG] [app] Starting request: urn:request:21a64260-50bc-4edc-9709-e17cec659315 (/api/v1/repository/build_test/bitbucket/build/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:07,354 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:07,354 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:07,355 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:07,365 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:07,367 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:07,374 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:07,375 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:07,380 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s', ['bitbucket', 'build_test', 3]) gunicorn-web stdout | 2023-01-17 06:19:07,385 [236] [DEBUG] [app] Ending request: urn:request:21a64260-50bc-4edc-9709-e17cec659315 (/api/v1/repository/build_test/bitbucket/build/) {'endpoint': 'api.repositorybuildlist', 'request_id': 'urn:request:21a64260-50bc-4edc-9709-e17cec659315', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/build/?limit=3', 'path': '/api/v1/repository/build_test/bitbucket/build/', 'parameters': {'limit': '3'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:07,386 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:07,386 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:07 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:07 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?namespace=build_test" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.033 1146 0.034) storagereplication stdout | 2023-01-17 06:19:07,686 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:19:07,699 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:19:08,130 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:19:08,553 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:19:08,583 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:19:08,783 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-web stdout | 2023-01-17 06:19:08,963 [236] [DEBUG] [app] Starting request: urn:request:97e514fb-aa38-4e6c-883f-43ee0bc5c136 (/api/v1/repository/build_test/bitbucket/trigger/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:08,964 [231] [DEBUG] [app] Starting request: urn:request:06b4751f-418a-4998-b810-43255d266812 (/api/v1/repository/build_test/bitbucket/build/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:08,964 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,964 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,964 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,964 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,964 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:08,964 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:08,974 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:08,974 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:08,974 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:08,975 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,975 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:08,975 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,975 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:08,975 [231] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:08,975 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,975 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:08,975 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,975 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,975 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,975 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,975 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,975 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,977 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:08,977 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:08,984 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,984 [231] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:08,985 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:08,985 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:08,990 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") WHERE (("t4"."username" = %s) AND ("t3"."name" = %s))', ['build_test', 'bitbucket']) gunicorn-web stdout | 2023-01-17 06:19:08,991 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s', ['bitbucket', 'build_test', 10]) gunicorn-web stdout | 2023-01-17 06:19:08,994 [236] [DEBUG] [app] Ending request: urn:request:97e514fb-aa38-4e6c-883f-43ee0bc5c136 (/api/v1/repository/build_test/bitbucket/trigger/) {'endpoint': 'api.buildtriggerlist', 'request_id': 'urn:request:97e514fb-aa38-4e6c-883f-43ee0bc5c136', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/', 'path': '/api/v1/repository/build_test/bitbucket/trigger/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:08,995 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:08,995 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:08 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/1.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:08 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/2.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.033 513 0.032) gunicorn-web stdout | 2023-01-17 06:19:08,996 [231] [DEBUG] [app] Ending request: urn:request:06b4751f-418a-4998-b810-43255d266812 (/api/v1/repository/build_test/bitbucket/build/) {'endpoint': 'api.repositorybuildlist', 'request_id': 'urn:request:06b4751f-418a-4998-b810-43255d266812', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/build/?limit=10', 'path': '/api/v1/repository/build_test/bitbucket/build/', 'parameters': {'limit': '10'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:08,997 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:08,997 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:08 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=10 HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:08 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=10 HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.035 97 0.035) repositoryactioncounter stdout | 2023-01-17 06:19:09,037 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:19:09,498 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:19:10,057 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:19:10,166 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 06:19:10,212 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:10,224 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:10,224 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:10,235 [60] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 06:19:10,429 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:19:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:19:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:40.738568+00:00 (in 29.999569 seconds) buildlogsarchiver stdout | 2023-01-17 06:19:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:40 UTC)" (scheduled at 2023-01-17 06:19:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:19:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 19, 10, 739415), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:19:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:19:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:19:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:40 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:11,490 [236] [DEBUG] [app] Starting request: urn:request:6d482451-83f6-4a40-9899-3251c89d5263 (/bitbucket/setup/build_test/bitbucket) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:11,490 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:11,490 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:11,490 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:11,501 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:11,501 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:11,501 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:11,501 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:11,501 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:11,501 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:11,503 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:11,510 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:11,511 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:11,514 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:11,517 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "buildtriggerservice" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['bitbucket', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:11,521 [236] [DEBUG] [peewee] ('INSERT INTO "repositorybuildtrigger" ("uuid", "service_id", "repository_id", "connected_user_id", "secure_auth_token", "fully_migrated", "config", "pull_robot_id", "enabled", "disabled_datetime", "successive_failure_count", "successive_internal_error_count") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "repositorybuildtrigger"."id"', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 3, 1, 2, None, False, '{}', None, True, datetime.datetime(2023, 1, 17, 6, 19, 11, 521115), 0, 0]) gunicorn-web stdout | 2023-01-17 06:19:11,528 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:11,528 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:11,528 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '154706224279140313391673936351'), ('oauth_timestamp', '1673936351'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_callback', 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/oauth1/bitbucket/callback/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44')] gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_callback=https%3A%2F%2Fquayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com%2Foauth1%2Fbitbucket%2Fcallback%2Ftrigger%2F03697f15-5e57-4d5a-ae56-cb0fcc4b4f44&oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=154706224279140313391673936351&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936351&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/oauth/request_token/ gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: POST&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Foauth%2Frequest_token%2F&oauth_callback%3Dhttps%253A%252F%252Fquayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com%252Foauth1%252Fbitbucket%252Fcallback%252Ftrigger%252F03697f15-5e57-4d5a-ae56-cb0fcc4b4f44%26oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D154706224279140313391673936351%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936351%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: Oei48RU0zFck2qRLxZJaFMxyqY8= gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/oauth/request_token/ gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'User-Agent': b'python-requests/2.27.1', b'Accept-Encoding': b'gzip, deflate', b'Accept': b'*/*', b'Connection': b'keep-alive', b'Content-Length': b'0', b'Authorization': b'OAuth oauth_nonce="154706224279140313391673936351", oauth_timestamp="1673936351", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_callback="https%3A%2F%2Fquayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com%2Foauth1%2Fbitbucket%2Fcallback%2Ftrigger%2F03697f15-5e57-4d5a-ae56-cb0fcc4b4f44", oauth_signature="Oei48RU0zFck2qRLxZJaFMxyqY8%3D"'} gunicorn-web stdout | 2023-01-17 06:19:11,529 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:11,530 [236] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:11,635 [236] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "POST /!api/1.0/oauth/request_token/ HTTP/1.1" 200 112 gunicorn-web stdout | 2023-01-17 06:19:11,637 [236] [DEBUG] [peewee] ('UPDATE "repositorybuildtrigger" SET "uuid" = %s, "service_id" = %s, "repository_id" = %s, "connected_user_id" = %s, "secure_auth_token" = %s, "fully_migrated" = %s, "config" = %s, "pull_robot_id" = %s, "enabled" = %s, "disabled_datetime" = %s, "successive_failure_count" = %s, "successive_internal_error_count" = %s WHERE ("repositorybuildtrigger"."id" = %s)', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 3, 1, 2, 'v0$$5G/qBEeFnvBKDs0UFxLyXf/7VemPuJe8Ny+w4BJFvXXL8Dstext3gkJPXERqr9DIQCOVnmZuN+2vTuJwVw==', False, '{"access_token": "k2c9Y5Tu5qefqKvCug"}', None, True, datetime.datetime(2023, 1, 17, 6, 19, 11, 521115), 0, 0, 1]) gunicorn-web stdout | 2023-01-17 06:19:11,643 [236] [DEBUG] [app] Ending request: urn:request:6d482451-83f6-4a40-9899-3251c89d5263 (/bitbucket/setup/build_test/bitbucket) {'endpoint': 'web.attach_bitbucket_trigger', 'request_id': 'urn:request:6d482451-83f6-4a40-9899-3251c89d5263', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/bitbucket/setup/build_test/bitbucket', 'path': '/bitbucket/setup/build_test/bitbucket', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:11,644 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:11,644 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:11 +0000] "GET /bitbucket/setup/build_test/bitbucket HTTP/1.0" 302 367 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:11 +0000] "GET /bitbucket/setup/build_test/bitbucket HTTP/2.0" 302 367 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.156 561 0.155) gunicorn-web stdout | 2023-01-17 06:19:15,199 [231] [DEBUG] [app] Starting request: urn:request:4340fb66-67e1-4c9f-9812-ed7f22897412 (/oauth1/bitbucket/callback/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:15,200 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:15,200 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:15,200 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:15,211 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:15,211 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:15,212 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:15,217 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:15,221 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:15,221 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:15,221 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '162916765461991250201673936355'), ('oauth_timestamp', '1673936355'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'k2c9Y5Tu5qefqKvCug'), ('oauth_verifier', '4663141559')] gunicorn-web stdout | 2023-01-17 06:19:15,221 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=162916765461991250201673936355&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936355&oauth_token=k2c9Y5Tu5qefqKvCug&oauth_verifier=4663141559&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:15,221 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/oauth/access_token/ gunicorn-web stdout | 2023-01-17 06:19:15,222 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: POST&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Foauth%2Faccess_token%2F&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D162916765461991250201673936355%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936355%26oauth_token%3Dk2c9Y5Tu5qefqKvCug%26oauth_verifier%3D4663141559%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:15,222 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: vedp48hPgIwlp1e498+vAAtdq7I= gunicorn-web stdout | 2023-01-17 06:19:15,222 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:15,222 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/oauth/access_token/ gunicorn-web stdout | 2023-01-17 06:19:15,222 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'User-Agent': b'python-requests/2.27.1', b'Accept-Encoding': b'gzip, deflate', b'Accept': b'*/*', b'Connection': b'keep-alive', b'Content-Length': b'0', b'Authorization': b'OAuth oauth_nonce="162916765461991250201673936355", oauth_timestamp="1673936355", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="k2c9Y5Tu5qefqKvCug", oauth_verifier="4663141559", oauth_signature="vedp48hPgIwlp1e498%2BvAAtdq7I%3D"'} gunicorn-web stdout | 2023-01-17 06:19:15,222 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:15,223 [231] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:15,607 [231] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "POST /!api/1.0/oauth/access_token/ HTTP/1.1" 200 82 gunicorn-web stdout | 2023-01-17 06:19:15,609 [231] [DEBUG] [peewee] ('UPDATE "repositorybuildtrigger" SET "uuid" = %s, "service_id" = %s, "repository_id" = %s, "connected_user_id" = %s, "secure_auth_token" = %s, "secure_private_key" = %s, "fully_migrated" = %s, "config" = %s, "write_token_id" = %s, "pull_robot_id" = %s, "enabled" = %s, "disabled_reason_id" = %s, "disabled_datetime" = %s, "successive_failure_count" = %s, "successive_internal_error_count" = %s WHERE ("repositorybuildtrigger"."id" = %s)', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 3, 1, 2, 'v0$$7TkjbRQsJ/L9o9Q+AEH2ECW00b2CCSOzyrYuZGyeCDRWRuRCE6NLD0uLPAAmJVIPMdDAFxj/usUu6nCVs4hxvdxYorkxEvkXEvILg7hd0Mw=', None, False, '{"access_token": "k2c9Y5Tu5qefqKvCug"}', None, None, True, None, datetime.datetime(2023, 1, 17, 6, 19, 11, 521115), 0, 0, 1]) gunicorn-web stdout | 2023-01-17 06:19:15,615 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:15,615 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:15,615 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '93871725311031690861673936355'), ('oauth_timestamp', '1673936355'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:15,615 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=93871725311031690861673936355&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936355&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:15,615 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/user gunicorn-web stdout | 2023-01-17 06:19:15,616 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Fuser&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D93871725311031690861673936355%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936355%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:15,616 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: aeglji7h+ldvvu+xaJe/ePYNSJo= gunicorn-web stdout | 2023-01-17 06:19:15,616 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:15,616 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/user gunicorn-web stdout | 2023-01-17 06:19:15,616 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="93871725311031690861673936355", oauth_timestamp="1673936355", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="aeglji7h%2Bldvvu%2BxaJe%2FePYNSJo%3D"'} gunicorn-web stdout | 2023-01-17 06:19:15,616 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:15,617 [231] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 notificationworker stdout | 2023-01-17 06:19:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:25.646354+00:00 (in 9.999528 seconds) notificationworker stdout | 2023-01-17 06:19:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:25 UTC)" (scheduled at 2023-01-17 06:19:15.646354+00:00) notificationworker stdout | 2023-01-17 06:19:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 15, 647201), True, datetime.datetime(2023, 1, 17, 6, 19, 15, 647201), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:25 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:15,772 [231] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/user HTTP/1.1" 200 1242 gunicorn-web stdout | 2023-01-17 06:19:15,773 [231] [DEBUG] [peewee] ('UPDATE "repositorybuildtrigger" SET "uuid" = %s, "service_id" = %s, "repository_id" = %s, "connected_user_id" = %s, "secure_auth_token" = %s, "secure_private_key" = %s, "fully_migrated" = %s, "config" = %s, "write_token_id" = %s, "pull_robot_id" = %s, "enabled" = %s, "disabled_reason_id" = %s, "disabled_datetime" = %s, "successive_failure_count" = %s, "successive_internal_error_count" = %s WHERE ("repositorybuildtrigger"."id" = %s)', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 3, 1, 2, 'v0$$Qfg8MZFOKM+xOLm9xvZLIW6aMVKRSI+H+MLCZcqIFlYN2xIU7YbfUeDfSodJBYgSwuxsVRKeUYFVg0HWstprtAlgMNkqre7V5QYWeC3LuWU=', None, False, '{"access_token": "k2c9Y5Tu5qefqKvCug", "account_id": "6273d885f42962006fe0871f"}', None, None, True, None, datetime.datetime(2023, 1, 17, 6, 19, 11, 521115), 0, 0, 1]) gunicorn-web stdout | 2023-01-17 06:19:15,782 [231] [DEBUG] [peewee] ('UPDATE "repositorybuildtrigger" SET "uuid" = %s, "service_id" = %s, "repository_id" = %s, "connected_user_id" = %s, "secure_auth_token" = %s, "secure_private_key" = %s, "fully_migrated" = %s, "config" = %s, "write_token_id" = %s, "pull_robot_id" = %s, "enabled" = %s, "disabled_reason_id" = %s, "disabled_datetime" = %s, "successive_failure_count" = %s, "successive_internal_error_count" = %s WHERE ("repositorybuildtrigger"."id" = %s)', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 3, 1, 2, 'v0$$DNUePH52YwR2/Hmuv7Lbg9N+1H8lhYHL4SwJJMitBxFNKro+rMQVhDEpPDOgiedlmpx6+n+O5TKuqV3QuSxsXkWAqlIBb6W39wQH85P/nso=', None, False, '{"access_token": "k2c9Y5Tu5qefqKvCug", "account_id": "6273d885f42962006fe0871f", "nickname": "hu_weihua"}', None, None, True, None, datetime.datetime(2023, 1, 17, 6, 19, 11, 521115), 0, 0, 1]) gunicorn-web stdout | 2023-01-17 06:19:15,788 [231] [DEBUG] [endpoints.bitbuckettrigger] Redirecting to full url: /repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44 gunicorn-web stdout | 2023-01-17 06:19:15,788 [231] [DEBUG] [app] Ending request: urn:request:4340fb66-67e1-4c9f-9812-ed7f22897412 (/oauth1/bitbucket/callback/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44) {'endpoint': 'bitbuckettrigger.attach_bitbucket_build_trigger', 'request_id': 'urn:request:4340fb66-67e1-4c9f-9812-ed7f22897412', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/oauth1/bitbucket/callback/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44?oauth_verifier=4663141559&oauth_token=k2c9Y5Tu5qefqKvCug', 'path': '/oauth1/bitbucket/callback/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 'parameters': {'oauth_verifier': '4663141559', 'oauth_token': 'k2c9Y5Tu5qefqKvCug'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:15,789 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:15,789 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:15 +0000] "GET /oauth1/bitbucket/callback/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44?oauth_verifier=4663141559&oauth_token=k2c9Y5Tu5qefqKvCug HTTP/1.0" 302 361 "https://bitbucket.org/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:15 +0000] "GET /oauth1/bitbucket/callback/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44?oauth_verifier=4663141559&oauth_token=k2c9Y5Tu5qefqKvCug HTTP/2.0" 302 361 "https://bitbucket.org/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.591 636 0.591) gunicorn-web stdout | 2023-01-17 06:19:16,017 [231] [DEBUG] [app] Starting request: urn:request:9dab3122-4372-4819-9795-edff55a1d214 (/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:16,018 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:16,020 [231] [DEBUG] [app] Ending request: urn:request:9dab3122-4372-4819-9795-edff55a1d214 (/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44) {'endpoint': 'web.buildtrigger', 'request_id': 'urn:request:9dab3122-4372-4819-9795-edff55a1d214', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 'path': '/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:16,021 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:16 +0000] "GET /repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44 HTTP/1.0" 200 84905 "https://bitbucket.org/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44 HTTP/2.0" 200 13246 "https://bitbucket.org/" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.005 1302 0.004) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/font-awesome.css-a166e3c966c6.css HTTP/2.0" 200 8404 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 593 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/bootstrap.min.css-13f668494557.css HTTP/2.0" 200 24427 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 126 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/cssfamilySourceSansPro300400700-30cd793a4b9b.css HTTP/2.0" 200 224 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 75 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/bootstrap-datetimepicker.min.css-4b15a4ede750.css HTTP/2.0" 200 1470 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 75 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/cal-heatmap.css-904c3b9e8a60.css HTTP/2.0" 200 787 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 64 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/ng-tags-input.min.css-7782190f81cd.css HTTP/2.0" 200 1067 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 68 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/jquery.js-3c5c5966471e.js HTTP/2.0" 200 102774 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 70 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/bootstrap.min.js-0e773844b374.js HTTP/2.0" 200 11290 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 130 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/angular.min.js-f0f4228b82f7.js HTTP/2.0" 200 63479 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 63 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/angular-route.min.js-590b72b0eb88.js HTTP/2.0" 200 2236 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 67 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/angular-sanitize.min.js-ff589de0ec1b.js HTTP/2.0" 200 3116 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 69 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/angular-animate.min.js-636f7a51b2e1.js HTTP/2.0" 200 10296 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 69 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/angular-cookies.min.js-2455ea25d522.js HTTP/2.0" 200 814 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 68 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/momentjs-66a52705d33f.js HTTP/2.0" 200 19464 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 59 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/bootstrap-datepicker.min.js-3e3d8e141d5e.js HTTP/2.0" 200 9310 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 72 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/bootstrap-datetimepicker.min.js-6612bb423b71.js HTTP/2.0" 200 10819 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 75 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/bootbox4.1.0underscorejs1.5.2restangular1.2.0d3js3.3.3-6d47ddfef9c5.js HTTP/2.0" 200 71170 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.061 91 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/raven.min.js-ecd480647a39.js HTTP/2.0" 200 9492 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.061 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/cal-heatmap.min.js-8ffad728269a.js HTTP/2.0" 200 12991 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.061 66 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/angular-recaptcha.min.js-23859fca3602.js HTTP/2.0" 200 1781 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.061 70 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/ng-tags-input.min.js-fa3026ba8409.js HTTP/2.0" 200 6089 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.062 67 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/ldn/typeahead.bundle.min.js-716151ca4e5d.js HTTP/2.0" 200 16660 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.062 69 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:16 +0000] "GET /static/standalonelib/jquery.overscroll.min.js HTTP/2.0" 200 3479 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.187 66 -) securityworker stdout | 2023-01-17 06:19:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:19.537472+00:00 (in 2.600035 seconds) securityworker stdout | 2023-01-17 06:19:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:46 UTC)" (scheduled at 2023-01-17 06:19:16.936988+00:00) securityworker stdout | 2023-01-17 06:19:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:16,954 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:46 UTC)" executed successfully nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:17 +0000] "GET /static/build/main-quay-frontend-e932307d1657af47ffb2.bundle.js HTTP/2.0" 200 719574 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.787 78 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:17 +0000] "GET /static/img/RedHat.svg HTTP/2.0" 200 1233 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 105 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:18 +0000] "GET /static/build/83-quay-frontend.chunk.js HTTP/2.0" 200 1705 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:18 +0000] "GET /static/build/135-quay-frontend.chunk.js HTTP/2.0" 200 963 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:18 +0000] "GET /static/build/18-quay-frontend.chunk.js HTTP/2.0" 200 985 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 61 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:18 +0000] "GET /static/build/112-quay-frontend.chunk.js HTTP/2.0" 200 755 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:18 +0000] "GET /static/build/181-quay-frontend.chunk.js HTTP/2.0" 200 940 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:18 +0000] "GET /static/build/152-quay-frontend.chunk.js HTTP/2.0" 200 313 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 62 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:18 +0000] "GET /static/img/network-tile.png HTTP/2.0" 200 6289 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 53 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:19 +0000] "GET /static/build/489e5e00e1a9a80f88d55281894a87ff.woff HTTP/2.0" 200 7280 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 923 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:19 +0000] "GET /static/fonts/fontawesome-webfont.woff2?v=4.7.0 HTTP/2.0" 200 77160 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/static/ldn/font-awesome.css-a166e3c966c6.css" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 165 -) gunicorn-web stdout | 2023-01-17 06:19:19,145 [236] [DEBUG] [app] Starting request: urn:request:b6a20ad0-ab7a-4397-a691-a97489105c82 (/api/v1/messages) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:19,145 [231] [DEBUG] [app] Starting request: urn:request:4323ee3c-37fb-4918-86ce-ee0f5b59c452 (/api/v1/user/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:19,145 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,145 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,146 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,146 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,146 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,146 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:19 +0000] "GET /static/img/RH_Logo_Quay_Black_UX-horizontal.svg HTTP/2.0" 200 3210 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 69 -) gunicorn-web stdout | 2023-01-17 06:19:19,155 [233] [DEBUG] [app] Starting request: urn:request:ec269815-cc94-47ab-bb2c-761bd3e0e01b (/api/v1/repository/build_test/bitbucket) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:19,155 [235] [DEBUG] [app] Starting request: urn:request:331b949c-9684-4db0-9f66-8069fd6a2049 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:19,155 [233] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,156 [233] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,156 [235] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,156 [235] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,156 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,156 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,157 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:19,157 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:19,157 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:19,157 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,157 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:19,157 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,157 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,157 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,157 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,157 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,157 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."content", "t1"."uuid", "t1"."severity", "t1"."media_type_id", "t2"."id", "t2"."name" FROM "messages" AS "t1" INNER JOIN "mediatype" AS "t2" ON ("t1"."media_type_id" = "t2"."id")', []) gunicorn-web stdout | 2023-01-17 06:19:19,158 [231] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "team" AS "t2" ON ("t2"."organization_id" = "t1"."id") INNER JOIN "teammember" AS "t3" ON ("t3"."team_id" = "t2"."id") INNER JOIN "user" AS "t4" ON ("t4"."id" = "t3"."user_id") WHERE (("t1"."organization" = %s) AND ("t4"."username" = %s))', [True, 'user1']) gunicorn-web stdout | 2023-01-17 06:19:19,161 [236] [DEBUG] [app] Ending request: urn:request:b6a20ad0-ab7a-4397-a691-a97489105c82 (/api/v1/messages) {'endpoint': 'api.globalusermessages', 'request_id': 'urn:request:b6a20ad0-ab7a-4397-a691-a97489105c82', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/messages', 'path': '/api/v1/messages', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:19,162 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:19,162 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/messages HTTP/1.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/messages HTTP/2.0" 200 17 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.018 134 0.018) gunicorn-web stdout | 2023-01-17 06:19:19,163 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,163 [231] [DEBUG] [peewee] ('SELECT "t1"."service_ident", "t2"."name", "t1"."metadata_json" FROM "federatedlogin" AS "t1" INNER JOIN "loginservice" AS "t2" ON ("t1"."service_id" = "t2"."id") WHERE (("t2"."name" != %s) AND ("t1"."user_id" = %s))', ['quayrobot', 2]) gunicorn-web stdout | 2023-01-17 06:19:19,166 [233] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:19,166 [235] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:19,167 [233] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:19,167 [233] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,167 [235] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:19,167 [235] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,167 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."kind_id" FROM "userprompt" AS "t1" INNER JOIN "userpromptkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE ("t1"."user_id" = %s)', [2]) gunicorn-web stdout | 2023-01-17 06:19:19,167 [233] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:19,167 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,167 [233] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,167 [235] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:19,167 [233] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,167 [233] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,167 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,167 [235] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,167 [235] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,167 [235] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,169 [233] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:19,170 [235] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:19,170 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,174 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:19:19,176 [233] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,176 [233] [DEBUG] [endpoints.api.repository] Get repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:19,176 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,177 [235] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,177 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,178 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,180 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."repository_id", "t1"."created" FROM "star" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."user_id" = %s)) LIMIT %s OFFSET %s', [1, 2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,181 [231] [DEBUG] [peewee] ('INSERT INTO "repositorysize" ("repository_id", "size_bytes") SELECT "t1"."id", sum("t2"."layers_compressed_size") FROM "manifest" AS "t2" INNER JOIN "repository" AS "t1" ON ("t2"."repository_id" = "t1"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t2"."id") LEFT OUTER JOIN "repositorysize" AS "repositorysize" ON ("repositorysize"."repository_id" = "t1"."id") WHERE (("t1"."namespace_user_id" = %s) AND ("repositorysize"."repository_id" IS %s)) GROUP BY "t1"."id" RETURNING "repositorysize"."id"', [False, None, 1673936359180, 0, 2, None]) gunicorn-web stdout | 2023-01-17 06:19:19,183 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,184 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."count", "t1"."date" FROM "repositoryactioncount" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."date" >= %s))', [1, datetime.date(2022, 10, 17)]) gunicorn-web stdout | 2023-01-17 06:19:19,186 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,187 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,187 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,188 [233] [DEBUG] [app] Ending request: urn:request:ec269815-cc94-47ab-bb2c-761bd3e0e01b (/api/v1/repository/build_test/bitbucket) {'endpoint': 'api.repository', 'request_id': 'urn:request:ec269815-cc94-47ab-bb2c-761bd3e0e01b', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket?includeTags=false', 'path': '/api/v1/repository/build_test/bitbucket', 'parameters': {'includeTags': 'false'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:19,188 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/repository/build_test/bitbucket?includeTags=false HTTP/2.0" 200 342 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.034 76 0.034) gunicorn-web stdout | 2023-01-17 06:19:19,188 [233] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/repository/build_test/bitbucket?includeTags=false HTTP/1.0" 200 342 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:19:19,189 [235] [DEBUG] [app] Ending request: urn:request:331b949c-9684-4db0-9f66-8069fd6a2049 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44) {'endpoint': 'api.buildtrigger', 'request_id': 'urn:request:331b949c-9684-4db0-9f66-8069fd6a2049', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:19,189 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['user1']) gunicorn-web stdout | 2023-01-17 06:19:19,190 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44 HTTP/2.0" 200 316 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.036 95 0.036) gunicorn-web stdout | 2023-01-17 06:19:19,190 [235] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44 HTTP/1.0" 200 316 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:19:19,193 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['user1', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:19,196 [231] [DEBUG] [peewee] ('SELECT sum("t1"."layers_compressed_size") FROM "manifest" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN (SELECT "t3"."manifest_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."manifest_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."manifest_id" = "t1"."id") WHERE ("t2"."namespace_user_id" = %s)', [False, None, 1673936359196, 0, 2]) gunicorn-web stdout | 2023-01-17 06:19:19,200 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,200 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,201 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:19:19,204 [231] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,204 [231] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,205 [231] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:19:19,205 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,205 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:19,205 [231] [DEBUG] [app] Ending request: urn:request:4323ee3c-37fb-4918-86ce-ee0f5b59c452 (/api/v1/user/) {'endpoint': 'api.user', 'request_id': 'urn:request:4323ee3c-37fb-4918-86ce-ee0f5b59c452', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/', 'path': '/api/v1/user/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:19,206 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/user/ HTTP/2.0" 200 898 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.062 169 0.062) gunicorn-web stdout | 2023-01-17 06:19:19,206 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:19 +0000] "GET /api/v1/user/ HTTP/1.0" 200 898 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" repositorygcworker stdout | 2023-01-17 06:19:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:19:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:04.936835+00:00 (in 45.590224 seconds) repositorygcworker stdout | 2023-01-17 06:19:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:19 UTC)" (scheduled at 2023-01-17 06:19:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:19:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:19:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:19:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:46.936988+00:00 (in 27.399057 seconds) securityworker stdout | 2023-01-17 06:19:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:49 UTC)" (scheduled at 2023-01-17 06:19:19.537472+00:00) securityworker stdout | 2023-01-17 06:19:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:19:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:19:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:19:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:19:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:19:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:19:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:49 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:20,152 [235] [DEBUG] [app] Starting request: urn:request:8097102f-9dc2-4cfb-b524-f2ba48a7eb8c (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/namespaces) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:20,152 [235] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,152 [235] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,153 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:20,164 [235] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:20,164 [235] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:20,164 [235] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,164 [235] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:20,165 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,165 [235] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,165 [235] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,165 [235] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,167 [235] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:20,174 [235] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,175 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:20,180 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:20,185 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:20,188 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,188 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:20,188 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:20,189 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '183537628025093053691673936360'), ('oauth_timestamp', '1673936360'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:20,189 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=183537628025093053691673936360&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936360&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:20,189 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/user/repositories gunicorn-web stdout | 2023-01-17 06:19:20,189 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Fuser%2Frepositories&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D183537628025093053691673936360%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936360%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:20,189 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: v0syaEEAfa2Fu+4em2gCJZ6cHR8= gunicorn-web stdout | 2023-01-17 06:19:20,189 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:20,190 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/user/repositories gunicorn-web stdout | 2023-01-17 06:19:20,190 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="183537628025093053691673936360", oauth_timestamp="1673936360", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="v0syaEEAfa2Fu%2B4em2gCJZ6cHR8%3D"'} gunicorn-web stdout | 2023-01-17 06:19:20,190 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:20,191 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 builder stdout | 2023-01-17 06:19:20,245 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:20,256 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:20,257 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:20,267 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:20,281 [235] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/user/repositories HTTP/1.1" 200 755 gunicorn-web stdout | 2023-01-17 06:19:20,284 [235] [DEBUG] [app] Ending request: urn:request:8097102f-9dc2-4cfb-b524-f2ba48a7eb8c (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/namespaces) {'endpoint': 'api.buildtriggersourcenamespaces', 'request_id': 'urn:request:8097102f-9dc2-4cfb-b524-f2ba48a7eb8c', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/namespaces', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/namespaces', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:20,284 [235] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:20 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/namespaces HTTP/2.0" 200 322 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.134 449 0.133) gunicorn-web stdout | 2023-01-17 06:19:20,285 [235] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:20 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/namespaces HTTP/1.0" 200 322 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:19:20,744 [231] [DEBUG] [app] Starting request: urn:request:54b55ccb-e7a1-4a7a-a743-9ad6a517f3c4 (/api/v1/user/apptoken) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:20,745 [233] [DEBUG] [app] Starting request: urn:request:7153ffc7-73ce-4e1f-9d87-954d3c4ecbb1 (/api/v1/user/notifications) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:20,745 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,745 [233] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,745 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,745 [233] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,745 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:20,745 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:20,756 [233] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:20,756 [233] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:20,756 [233] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,756 [233] [DEBUG] [endpoints.api] Checking permission for user user1 gunicorn-web stdout | 2023-01-17 06:19:20,756 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,756 [233] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,756 [233] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,757 [233] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,758 [233] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."uuid", "t1"."kind_id", "t1"."metadata_json", "t1"."dismissed", "t1"."lookup_path", "t1"."created", "t1"."created" AS "cd", "t1"."target_id" FROM "notification" AS "t1" INNER JOIN "notificationkind" AS "t2" ON ("t1"."kind_id" = "t2"."id") WHERE (("t1"."dismissed" = %s) AND ("t1"."target_id" = %s))) UNION (SELECT "t3"."id", "t3"."uuid", "t3"."kind_id", "t3"."metadata_json", "t3"."dismissed", "t3"."lookup_path", "t3"."created", "t3"."created" AS "cd", "t3"."target_id" FROM "notification" AS "t3" INNER JOIN "notificationkind" AS "t4" ON ("t3"."kind_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t5"."id" = "t3"."target_id") INNER JOIN "team" AS "t6" ON ("t5"."id" = "t6"."organization_id") INNER JOIN "teamrole" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "teammember" AS "t8" ON ("t6"."id" = "t8"."team_id") INNER JOIN "user" AS "t9" ON ("t8"."user_id" = "t9"."id") WHERE (("t3"."dismissed" = %s) AND (("t9"."id" = %s) AND ("t7"."name" = %s)))) ORDER BY cd desc LIMIT %s', [False, 2, False, 2, 'admin', 6]) gunicorn-web stdout | 2023-01-17 06:19:20,759 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [endpoints.api] Checking permission for user user1 gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:20,760 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."uuid", "t1"."title", "t1"."token_name", "t1"."token_secret", "t1"."created", "t1"."expiration", "t1"."last_accessed" FROM "appspecificauthtoken" AS "t1" WHERE ((("t1"."user_id" = %s) AND ("t1"."expiration" <= %s)) AND ("t1"."expiration" > %s))', [2, datetime.datetime(2023, 1, 20, 1, 31, 20, 760521), datetime.datetime(2023, 1, 17, 6, 19, 20, 760558)]) gunicorn-web stdout | 2023-01-17 06:19:20,764 [231] [DEBUG] [app] Ending request: urn:request:54b55ccb-e7a1-4a7a-a743-9ad6a517f3c4 (/api/v1/user/apptoken) {'endpoint': 'api.apptokens', 'request_id': 'urn:request:54b55ccb-e7a1-4a7a-a743-9ad6a517f3c4', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/apptoken?expiring=true', 'path': '/api/v1/user/apptoken', 'parameters': {'expiring': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:20,764 [233] [DEBUG] [app] Ending request: urn:request:7153ffc7-73ce-4e1f-9d87-954d3c4ecbb1 (/api/v1/user/notifications) {'endpoint': 'api.usernotificationlist', 'request_id': 'urn:request:7153ffc7-73ce-4e1f-9d87-954d3c4ecbb1', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/user/notifications', 'path': '/api/v1/user/notifications', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:20,765 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:20,765 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:20 +0000] "GET /api/v1/user/apptoken?expiring=true HTTP/1.0" 200 38 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:19:20,765 [233] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:20 +0000] "GET /api/v1/user/apptoken?expiring=true HTTP/2.0" 200 38 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.022 61 0.022) gunicorn-web stdout | 2023-01-17 06:19:20,765 [233] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:20 +0000] "GET /api/v1/user/notifications HTTP/1.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:20 +0000] "GET /api/v1/user/notifications HTTP/2.0" 200 43 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.022 217 0.022) gcworker stdout | 2023-01-17 06:19:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:19:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:51.348605+00:00 (in 29.999522 seconds) gcworker stdout | 2023-01-17 06:19:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:51 UTC)" (scheduled at 2023-01-17 06:19:21.348605+00:00) gcworker stdout | 2023-01-17 06:19:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726761349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:19:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:19:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:19:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:19:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:22,059 [236] [DEBUG] [app] Starting request: urn:request:5df2c8d6-865d-4cf1-854d-636b0aebeac3 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:22,060 [231] [DEBUG] [app] Starting request: urn:request:5cbb873b-dd3b-4e96-a28c-99654677e482 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:22,064 [230] [DEBUG] [app] Starting request: urn:request:9381f12b-a592-49ce-95f4-79bd6e1108bd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:22,065 [243] [DEBUG] [app] Starting request: urn:request:1eb1e8ac-21a1-405c-a1a0-12a9acaa963e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:22,065 [230] [DEBUG] [app] Ending request: urn:request:9381f12b-a592-49ce-95f4-79bd6e1108bd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9381f12b-a592-49ce-95f4-79bd6e1108bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:19:22,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:19:22,065 [243] [DEBUG] [app] Ending request: urn:request:1eb1e8ac-21a1-405c-a1a0-12a9acaa963e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1eb1e8ac-21a1-405c-a1a0-12a9acaa963e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:19:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:19:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:22,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:22,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:22,069 [231] [DEBUG] [app] Starting request: urn:request:df14b6c0-e0c1-4a39-a728-7e83cda563a2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:22,069 [231] [DEBUG] [app] Ending request: urn:request:df14b6c0-e0c1-4a39-a728-7e83cda563a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df14b6c0-e0c1-4a39-a728-7e83cda563a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:22,070 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:22,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:22,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:22,071 [233] [DEBUG] [app] Starting request: urn:request:c55e06ab-984a-4e02-a61d-86a7de39d500 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:22,071 [233] [DEBUG] [app] Ending request: urn:request:c55e06ab-984a-4e02-a61d-86a7de39d500 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c55e06ab-984a-4e02-a61d-86a7de39d500', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:22,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:22,087 [236] [DEBUG] [app] Ending request: urn:request:5df2c8d6-865d-4cf1-854d-636b0aebeac3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5df2c8d6-865d-4cf1-854d-636b0aebeac3', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:22,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:22,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:19:22,087 [231] [DEBUG] [app] Ending request: urn:request:5cbb873b-dd3b-4e96-a28c-99654677e482 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5cbb873b-dd3b-4e96-a28c-99654677e482', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:19:23,753 [236] [DEBUG] [app] Starting request: urn:request:4aded7b4-9622-4c47-85f1-5d5ff8d62395 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/sources) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:23,753 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:23,753 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:23,754 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:23,764 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:23,764 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:23,764 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:23,765 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:23,765 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:23,765 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:23,765 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:23,765 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:23,767 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:23,774 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:23,775 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:23,779 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:23,784 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:23,789 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:23,792 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:23,792 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:23,792 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:23,792 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '59492779976100126061673936363'), ('oauth_timestamp', '1673936363'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:23,792 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=59492779976100126061673936363&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936363&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:23,792 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/user/repositories gunicorn-web stdout | 2023-01-17 06:19:23,793 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Fuser%2Frepositories&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D59492779976100126061673936363%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936363%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:23,793 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: 6Myx1SR3IkrFC2L/R6uRD8EQtA0= gunicorn-web stdout | 2023-01-17 06:19:23,793 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:23,793 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/user/repositories gunicorn-web stdout | 2023-01-17 06:19:23,793 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="59492779976100126061673936363", oauth_timestamp="1673936363", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="6Myx1SR3IkrFC2L%2FR6uRD8EQtA0%3D"'} gunicorn-web stdout | 2023-01-17 06:19:23,793 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:23,794 [236] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:23,878 [236] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/user/repositories HTTP/1.1" 200 755 gunicorn-web stdout | 2023-01-17 06:19:23,881 [236] [DEBUG] [app] Ending request: urn:request:4aded7b4-9622-4c47-85f1-5d5ff8d62395 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/sources) {'endpoint': 'api.buildtriggersources', 'request_id': 'urn:request:4aded7b4-9622-4c47-85f1-5d5ff8d62395', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/sources', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/sources', 'parameters': {}, 'json_body': {'namespace': '{35aa643d-9d7c-4c91-970f-1915ef761a21}'}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:23,881 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:23,882 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:23 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/sources HTTP/1.0" 200 296 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:23 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/sources HTTP/2.0" 200 296 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.139 530 0.129) notificationworker stdout | 2023-01-17 06:19:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:35.646354+00:00 (in 9.999512 seconds) notificationworker stdout | 2023-01-17 06:19:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:35 UTC)" (scheduled at 2023-01-17 06:19:25.646354+00:00) notificationworker stdout | 2023-01-17 06:19:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 25, 647273), True, datetime.datetime(2023, 1, 17, 6, 19, 25, 647273), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:19:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:19:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:35.853396+00:00 (in 6.889286 seconds) exportactionlogsworker stdout | 2023-01-17 06:19:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:28 UTC)" (scheduled at 2023-01-17 06:19:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:19:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:19:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:28 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:29,675 [236] [DEBUG] [app] Starting request: urn:request:2bc7b7a6-8a99-4e7c-9e9b-3114e8e2985b (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/subdir) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:29,675 [231] [DEBUG] [app] Starting request: urn:request:bd487c11-87fd-4105-aaa0-df254743e87c (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:29,675 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,675 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,676 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,676 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,676 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,676 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,687 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,687 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,689 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:29,689 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:29,696 [231] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,696 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:29,697 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,697 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,701 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,702 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,706 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,706 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,710 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '173475944036464561081673936369'), ('oauth_timestamp', '1673936369'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=173475944036464561081673936369&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936369&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches-tags gunicorn-web stdout | 2023-01-17 06:19:29,711 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fbranches-tags&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D173475944036464561081673936369%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936369%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: ZYQhMRrfPa46IPvP9cF+iXlHebI= gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches-tags gunicorn-web stdout | 2023-01-17 06:19:29,711 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="173475944036464561081673936369", oauth_timestamp="1673936369", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="ZYQhMRrfPa46IPvP9cF%2BiXlHebI%3D"'} gunicorn-web stdout | 2023-01-17 06:19:29,712 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:29,713 [231] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:29,714 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:29,715 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:29,715 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:29,715 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '152717168752142092241673936369'), ('oauth_timestamp', '1673936369'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:29,715 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=152717168752142092241673936369&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936369&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:29,715 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches gunicorn-web stdout | 2023-01-17 06:19:29,715 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fbranches&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D152717168752142092241673936369%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936369%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:29,715 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: cIsICTmoJduPITvLXpa9qL07mJ4= gunicorn-web stdout | 2023-01-17 06:19:29,716 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:29,716 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches gunicorn-web stdout | 2023-01-17 06:19:29,716 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="152717168752142092241673936369", oauth_timestamp="1673936369", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="cIsICTmoJduPITvLXpa9qL07mJ4%3D"'} gunicorn-web stdout | 2023-01-17 06:19:29,716 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:29,717 [236] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:29,806 [231] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches-tags HTTP/1.1" 200 178 gunicorn-web stdout | 2023-01-17 06:19:29,808 [231] [DEBUG] [app] Ending request: urn:request:bd487c11-87fd-4105-aaa0-df254743e87c (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs) {'endpoint': 'api.buildtriggerfieldvalues', 'request_id': 'urn:request:bd487c11-87fd-4105-aaa0-df254743e87c', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs', 'parameters': {}, 'json_body': {'build_source': '{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test'}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:29,809 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:29 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs HTTP/2.0" 200 51 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.558 1922 0.135) gunicorn-web stdout | 2023-01-17 06:19:29,809 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:29 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs HTTP/1.0" 200 51 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:19:29,903 [236] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches HTTP/1.1" 200 429 gunicorn-web stdout | 2023-01-17 06:19:29,905 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:29,905 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:29,905 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '14602793513130156921673936369'), ('oauth_timestamp', '1673936369'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=14602793513130156921673936369&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936369&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/src/master/ gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fsrc%2Fmaster%2F&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D14602793513130156921673936369%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936369%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: RgR3sEQLzanKcX6VveTe9iCooP0= gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/src/master/ gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="14602793513130156921673936369", oauth_timestamp="1673936369", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="RgR3sEQLzanKcX6VveTe9iCooP0%3D"'} gunicorn-web stdout | 2023-01-17 06:19:29,906 [236] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:29,907 [236] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 builder stdout | 2023-01-17 06:19:30,277 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:30,292 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:30,292 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:30,303 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:30,337 [236] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/src/master/ HTTP/1.1" 200 1268 gunicorn-web stdout | 2023-01-17 06:19:30,339 [236] [DEBUG] [app] Ending request: urn:request:2bc7b7a6-8a99-4e7c-9e9b-3114e8e2985b (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/subdir) {'endpoint': 'api.buildtriggersubdirs', 'request_id': 'urn:request:2bc7b7a6-8a99-4e7c-9e9b-3114e8e2985b', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/subdir', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/subdir', 'parameters': {}, 'json_body': {'build_source': '{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test'}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:30,340 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:30,340 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:30 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/subdir HTTP/1.0" 200 98 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:30 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/subdir HTTP/2.0" 200 98 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.666 176 0.666) gcworker stdout | 2023-01-17 06:19:31,339 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:19:31,380 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:19:31,749 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:19:31,755 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:19:31,758 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:19:31,761 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:19:31,763 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:19:32,336 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:19:32,757 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:19:32,811 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:19:32,844 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:19:33,285 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:19:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:45.646354+00:00 (in 9.999551 seconds) notificationworker stdout | 2023-01-17 06:19:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:45 UTC)" (scheduled at 2023-01-17 06:19:35.646354+00:00) notificationworker stdout | 2023-01-17 06:19:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 35, 647195), True, datetime.datetime(2023, 1, 17, 6, 19, 35, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:19:35,729 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:19:35,731 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:19:35,734 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:19:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:19:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:28.963662+00:00 (in 53.109781 seconds) exportactionlogsworker stdout | 2023-01-17 06:19:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:35 UTC)" (scheduled at 2023-01-17 06:19:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:19:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:19:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 35, 854262), True, datetime.datetime(2023, 1, 17, 6, 19, 35, 854262), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:19:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:19:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:19:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:19:36,229 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:19:36,234 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:19:36,237 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:19:36,248 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:19:36,251 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:19:36,254 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:19:36,257 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:19:36,262 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:19:36,265 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:19:36,489 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:19:36,601 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:19:37,059 [233] [DEBUG] [app] Starting request: urn:request:99f8d907-822c-4b2a-9173-4e7ae1384b07 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:37,059 [236] [DEBUG] [app] Starting request: urn:request:ac517f2c-5ff7-4c5b-9411-6d80a703f110 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:37,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:37,065 [230] [DEBUG] [app] Starting request: urn:request:bd4c4dbd-a83c-4cbc-b3db-7f1c14b0dc97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:37,065 [240] [DEBUG] [app] Starting request: urn:request:2783fdc0-31e8-4fd3-893f-761ea59a827a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:37,065 [230] [DEBUG] [app] Ending request: urn:request:bd4c4dbd-a83c-4cbc-b3db-7f1c14b0dc97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd4c4dbd-a83c-4cbc-b3db-7f1c14b0dc97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:19:37,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:19:37,065 [240] [DEBUG] [app] Ending request: urn:request:2783fdc0-31e8-4fd3-893f-761ea59a827a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2783fdc0-31e8-4fd3-893f-761ea59a827a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:37,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:19:37,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:19:37,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:37,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:37,070 [235] [DEBUG] [app] Starting request: urn:request:1b040ef7-1be1-4a1b-aee0-2fd2bc42ba04 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:37,070 [231] [DEBUG] [app] Starting request: urn:request:609826b7-da8e-41fb-a58d-b944249b78e9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:37,070 [235] [DEBUG] [app] Ending request: urn:request:1b040ef7-1be1-4a1b-aee0-2fd2bc42ba04 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1b040ef7-1be1-4a1b-aee0-2fd2bc42ba04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:37,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:37,070 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:19:37,071 [231] [DEBUG] [app] Ending request: urn:request:609826b7-da8e-41fb-a58d-b944249b78e9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:609826b7-da8e-41fb-a58d-b944249b78e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:37,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.002) gunicorn-web stdout | 2023-01-17 06:19:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:37,077 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:37,086 [233] [DEBUG] [app] Ending request: urn:request:99f8d907-822c-4b2a-9173-4e7ae1384b07 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:99f8d907-822c-4b2a-9173-4e7ae1384b07', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:37,086 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:37,087 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) gunicorn-web stdout | 2023-01-17 06:19:37,087 [236] [DEBUG] [app] Ending request: urn:request:ac517f2c-5ff7-4c5b-9411-6d80a703f110 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac517f2c-5ff7-4c5b-9411-6d80a703f110', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 06:19:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:19:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:37.537407+00:00 (in 0.280331 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:19:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:37 UTC)" (scheduled at 2023-01-17 06:19:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:19:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:19:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 37, 257475), True, datetime.datetime(2023, 1, 17, 6, 19, 37, 257475), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:19:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:19:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:19:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:19:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:19:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:37.256652+00:00 (in 59.718844 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:19:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:37 UTC)" (scheduled at 2023-01-17 06:19:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:19:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:19:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:19:37,722 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:19:37,732 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:19:38,166 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:19:38,569 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:19:38,589 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} gunicorn-web stdout | 2023-01-17 06:19:38,754 [235] [DEBUG] [app] Starting request: urn:request:bfdb2b03-cc82-4234-a897-28adacd1777f (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:38,754 [235] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:38,754 [235] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:38,755 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:38,765 [235] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:38,768 [235] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:38,774 [235] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:38,775 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:38,780 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:38,785 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:38,789 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:38,790 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:19:38,794 [235] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:19:38,794 [235] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:19:38,794 [235] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:19:38,794 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:38,794 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '90628710660238223731673936378'), ('oauth_timestamp', '1673936378'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=90628710660238223731673936378&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936378&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/raw/master/Dockerfile gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fraw%2Fmaster%2FDockerfile&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D90628710660238223731673936378%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936378%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: JwHlVS7IMbLv/5G36xiMJMaxiBw= gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/raw/master/Dockerfile gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="90628710660238223731673936378", oauth_timestamp="1673936378", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="JwHlVS7IMbLv%2F5G36xiMJMaxiBw%3D"'} gunicorn-web stdout | 2023-01-17 06:19:38,795 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:38,796 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 securityscanningnotificationworker stdout | 2023-01-17 06:19:38,806 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} gunicorn-web stdout | 2023-01-17 06:19:38,958 [235] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/raw/master/Dockerfile HTTP/1.1" 200 513 gunicorn-web stdout | 2023-01-17 06:19:38,960 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed", "t2"."id", "t2"."robot_account_id", "t2"."token", "t2"."fully_migrated", "t3"."id", "t3"."robot_account_id", "t3"."description", "t3"."unstructured_json" FROM "user" AS "t1" LEFT OUTER JOIN "robotaccountmetadata" AS "t3" ON ("t3"."robot_account_id" = "t1"."id") INNER JOIN "robotaccounttoken" AS "t2" ON ("t2"."robot_account_id" = "t1"."id") WHERE (("t1"."robot" = %s) AND ("t1"."username" ILIKE %s))', [True, 'build_test+%']) gunicorn-web stdout | 2023-01-17 06:19:38,964 [235] [DEBUG] [app] Ending request: urn:request:bfdb2b03-cc82-4234-a897-28adacd1777f (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze) {'endpoint': 'api.buildtriggeranalyze', 'request_id': 'urn:request:bfdb2b03-cc82-4234-a897-28adacd1777f', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze', 'parameters': {}, 'json_body': {'config': {'build_source': '{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test', 'dockerfile_path': '/Dockerfile', 'context': None}}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:38,965 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:38,965 [235] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:38 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze HTTP/1.0" 200 115 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:38 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze HTTP/2.0" 200 115 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.213 1978 0.213) repositoryactioncounter stdout | 2023-01-17 06:19:39,065 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:19:39,506 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:19:40,093 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:19:40,202 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 06:19:40,313 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:40,325 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:40,325 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:40,336 [60] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 06:19:40,464 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:19:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:19:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:10.738568+00:00 (in 29.999553 seconds) buildlogsarchiver stdout | 2023-01-17 06:19:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:10 UTC)" (scheduled at 2023-01-17 06:19:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:19:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 19, 40, 739418), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:19:40,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:19:40,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:19:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:19:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:19:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:42.941536+00:00 (in 1.792821 seconds) namespacegcworker stdout | 2023-01-17 06:19:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:41 UTC)" (scheduled at 2023-01-17 06:19:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:19:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:19:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 41, 149106), True, datetime.datetime(2023, 1, 17, 6, 19, 41, 149106), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:19:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:19:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:19:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:41 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:41,858 [235] [DEBUG] [app] Starting request: urn:request:7511fc90-3ac0-47de-bde8-58d221ae3ce9 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:19:41,858 [235] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:41,858 [235] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:41,859 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:41,870 [235] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:41,872 [235] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:41,879 [235] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:41,880 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:41,885 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:41,890 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:41,894 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:41,895 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed", "t3"."id", "t3"."name" FROM "team" AS "t1" INNER JOIN "teamrole" AS "t3" ON ("t1"."role_id" = "t3"."id") INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") INNER JOIN "teammember" AS "t4" ON ("t4"."team_id" = "t1"."id") INNER JOIN "user" AS "t5" ON ("t4"."user_id" = "t5"."id") WHERE (("t5"."id" = %s) AND ("t2"."organization" = %s))', [2, True]) gunicorn-web stdout | 2023-01-17 06:19:41,899 [235] [DEBUG] [auth.permissions] Organization team added permission: _NamespaceWideNeed(type='organization', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:19:41,899 [235] [DEBUG] [auth.permissions] Organization team added repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='build_test', role='admin') gunicorn-web stdout | 2023-01-17 06:19:41,899 [235] [DEBUG] [auth.permissions] Team added permission: _TeamTypeNeed(type='orgteam', orgname='build_test', teamname='owners', role='admin') gunicorn-web stdout | 2023-01-17 06:19:41,899 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:41,899 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:41,899 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '103683659871061627651673936381'), ('oauth_timestamp', '1673936381'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=103683659871061627651673936381&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936381&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/raw/master/Dockerfile gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fraw%2Fmaster%2FDockerfile&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D103683659871061627651673936381%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936381%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: cceu7OJgr5VrSmW2PDPiCx1SYSg= gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/raw/master/Dockerfile gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="103683659871061627651673936381", oauth_timestamp="1673936381", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="cceu7OJgr5VrSmW2PDPiCx1SYSg%3D"'} gunicorn-web stdout | 2023-01-17 06:19:41,900 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:19:41,901 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:42,040 [235] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/raw/master/Dockerfile HTTP/1.1" 200 513 gunicorn-web stdout | 2023-01-17 06:19:42,043 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed", "t2"."id", "t2"."robot_account_id", "t2"."token", "t2"."fully_migrated", "t3"."id", "t3"."robot_account_id", "t3"."description", "t3"."unstructured_json" FROM "user" AS "t1" LEFT OUTER JOIN "robotaccountmetadata" AS "t3" ON ("t3"."robot_account_id" = "t1"."id") INNER JOIN "robotaccounttoken" AS "t2" ON ("t2"."robot_account_id" = "t1"."id") WHERE (("t1"."robot" = %s) AND ("t1"."username" ILIKE %s))', [True, 'build_test+%']) gunicorn-web stdout | 2023-01-17 06:19:42,047 [235] [DEBUG] [app] Ending request: urn:request:7511fc90-3ac0-47de-bde8-58d221ae3ce9 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze) {'endpoint': 'api.buildtriggeranalyze', 'request_id': 'urn:request:7511fc90-3ac0-47de-bde8-58d221ae3ce9', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze', 'parameters': {}, 'json_body': {'config': {'build_source': '{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test', 'dockerfile_path': '/Dockerfile', 'context': '/'}}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:42,047 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:42,048 [235] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:19:42 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze HTTP/1.0" 200 115 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:19:42 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/analyze HTTP/2.0" 200 115 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.235 584 0.191) namespacegcworker stdout | 2023-01-17 06:19:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:19:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:41.148245+00:00 (in 58.206227 seconds) namespacegcworker stdout | 2023-01-17 06:19:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:42 UTC)" (scheduled at 2023-01-17 06:19:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:19:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:19:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:20:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:19:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:55.646354+00:00 (in 9.999575 seconds) notificationworker stdout | 2023-01-17 06:19:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:55 UTC)" (scheduled at 2023-01-17 06:19:45.646354+00:00) notificationworker stdout | 2023-01-17 06:19:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 45, 647266), True, datetime.datetime(2023, 1, 17, 6, 19, 45, 647266), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:19:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:19:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:19:49.537472+00:00 (in 2.600015 seconds) securityworker stdout | 2023-01-17 06:19:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:16 UTC)" (scheduled at 2023-01-17 06:19:46.936988+00:00) securityworker stdout | 2023-01-17 06:19:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:16 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:47,516 [231] [DEBUG] [app] Starting request: urn:request:cda4f162-ba57-477c-b51c-a5ce9327e183 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/activate) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:19:47,516 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:47,516 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:47,517 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:47,527 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:47,529 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:47,536 [231] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:47,537 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:47,541 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:47,546 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:47,551 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:47,554 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:47,554 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "role" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['write', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:47,558 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:47,562 [231] [DEBUG] [peewee] ('INSERT INTO "accesstoken" ("friendly_name", "token_name", "token_code", "repository_id", "created", "role_id", "temporary") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "accesstoken"."id"', ['Build Trigger: bitbucket', 'BNSWIICZZWZHFS7YQC0MH6GWMQCB85IE', 'v0$$8bPgE2k4DHF/6jLO/fy3TfpwS1C9HcTCeUu9au800qAvPiKdU/q+CNoYuZk0XxbcgrMTRrvOfzpIyZGs7Q==', 1, datetime.datetime(2023, 1, 17, 6, 19, 47, 561818), 2, False]) gunicorn-web stdout | 2023-01-17 06:19:47,596 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:47,596 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: True gunicorn-web stdout | 2023-01-17 06:19:47,596 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '154086058606217900101673936387'), ('oauth_timestamp', '1673936387'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM'), ('label', 'Red Hat Quay webhook key'), ('key', 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGiUMaQPU6mKGhlyf8dPJliJ0qfN2zTWFFq15BzEt44RqlZCKmIqx2fe5AJba4QHRQzboT5SDBw3Zk5tRtyukpC+5JNZ1IMPgV2j1GH0wP9L0hDa1L1BIB8GEXu/afw7TQ/STyHCOoAMIjoZuOXJA/zsUtWs6s/fOTdrHQJndFViiR7miyOYOspAJ2pXsUlxu6sAkWZRyNv8QI8388/uwrSpWI4quNno6+ATmsnuAcM5xQAmwYnZeFGKbMoqHAt3Qq0Az4iGVpm/yn+utHKdPoIe6Flf4HD1pEtPQlIW7HzMxnOCpz01o6z17CfvhUoxusAmqbIGRPQL4gGR2NJnzF')] gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: key=ssh-rsa%20AAAAB3NzaC1yc2EAAAADAQABAAABAQDGiUMaQPU6mKGhlyf8dPJliJ0qfN2zTWFFq15BzEt44RqlZCKmIqx2fe5AJba4QHRQzboT5SDBw3Zk5tRtyukpC%2B5JNZ1IMPgV2j1GH0wP9L0hDa1L1BIB8GEXu%2Fafw7TQ%2FSTyHCOoAMIjoZuOXJA%2FzsUtWs6s%2FfOTdrHQJndFViiR7miyOYOspAJ2pXsUlxu6sAkWZRyNv8QI8388%2FuwrSpWI4quNno6%2BATmsnuAcM5xQAmwYnZeFGKbMoqHAt3Qq0Az4iGVpm%2Fyn%2ButHKdPoIe6Flf4HD1pEtPQlIW7HzMxnOCpz01o6z17CfvhUoxusAmqbIGRPQL4gGR2NJnzF&label=Red%20Hat%20Quay%20webhook%20key&oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=154086058606217900101673936387&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936387&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/deploy-keys gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: POST&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fdeploy-keys&key%3Dssh-rsa%2520AAAAB3NzaC1yc2EAAAADAQABAAABAQDGiUMaQPU6mKGhlyf8dPJliJ0qfN2zTWFFq15BzEt44RqlZCKmIqx2fe5AJba4QHRQzboT5SDBw3Zk5tRtyukpC%252B5JNZ1IMPgV2j1GH0wP9L0hDa1L1BIB8GEXu%252Fafw7TQ%252FSTyHCOoAMIjoZuOXJA%252FzsUtWs6s%252FfOTdrHQJndFViiR7miyOYOspAJ2pXsUlxu6sAkWZRyNv8QI8388%252FuwrSpWI4quNno6%252BATmsnuAcM5xQAmwYnZeFGKbMoqHAt3Qq0Az4iGVpm%252Fyn%252ButHKdPoIe6Flf4HD1pEtPQlIW7HzMxnOCpz01o6z17CfvhUoxusAmqbIGRPQL4gGR2NJnzF%26label%3DRed%2520Hat%2520Quay%2520webhook%2520key%26oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D154086058606217900101673936387%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936387%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: ApFtTp/ghWfwWiLxGulOkdJGUjA= gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/deploy-keys gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Content-Length': b'433', b'Content-Type': b'application/x-www-form-urlencoded', b'Authorization': b'OAuth oauth_nonce="154086058606217900101673936387", oauth_timestamp="1673936387", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="ApFtTp%2FghWfwWiLxGulOkdJGUjA%3D"'} gunicorn-web stdout | 2023-01-17 06:19:47,597 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: b'label=Red+Hat+Quay+webhook+key&key=ssh-rsa+AAAAB3NzaC1yc2EAAAADAQABAAABAQDGiUMaQPU6mKGhlyf8dPJliJ0qfN2zTWFFq15BzEt44RqlZCKmIqx2fe5AJba4QHRQzboT5SDBw3Zk5tRtyukpC%2B5JNZ1IMPgV2j1GH0wP9L0hDa1L1BIB8GEXu%2Fafw7TQ%2FSTyHCOoAMIjoZuOXJA%2FzsUtWs6s%2FfOTdrHQJndFViiR7miyOYOspAJ2pXsUlxu6sAkWZRyNv8QI8388%2FuwrSpWI4quNno6%2BATmsnuAcM5xQAmwYnZeFGKbMoqHAt3Qq0Az4iGVpm%2Fyn%2ButHKdPoIe6Flf4HD1pEtPQlIW7HzMxnOCpz01o6z17CfvhUoxusAmqbIGRPQL4gGR2NJnzF' gunicorn-web stdout | 2023-01-17 06:19:47,598 [231] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:49,063 [231] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "POST /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/deploy-keys HTTP/1.1" 200 443 gunicorn-web stdout | 2023-01-17 06:19:49,065 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:19:49,065 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:19:49,065 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '158935838660174704061673936389'), ('oauth_timestamp', '1673936389'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:19:49,065 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=158935838660174704061673936389&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936389&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:19:49,065 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://api.bitbucket.org/2.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/hooks gunicorn-web stdout | 2023-01-17 06:19:49,066 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: POST&https%3A%2F%2Fapi.bitbucket.org%2F2.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fhooks&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D158935838660174704061673936389%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936389%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:19:49,066 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: 6JmWTDPG68FFcb8W3K7boR8WLOg= gunicorn-web stdout | 2023-01-17 06:19:49,066 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:19:49,066 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://api.bitbucket.org/2.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/hooks gunicorn-web stdout | 2023-01-17 06:19:49,066 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Content-Type': b'application/json', b'Content-Length': b'369', b'Authorization': b'OAuth oauth_nonce="158935838660174704061673936389", oauth_timestamp="1673936389", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="6JmWTDPG68FFcb8W3K7boR8WLOg%3D"'} gunicorn-web stdout | 2023-01-17 06:19:49,066 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: '{"description": "Webhook for invoking builds on Red Hat Quay", "url": "https://$token:BNSWIICZZWZHFS7YQC0MH6GWMQCB85IENBSDQ6HWN16DZ7DTMI2MG7FAFGQDP1A7@quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/webhooks/push/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44", "active": true, "events": ["repo:push"], "skip_cert_verification": false}' gunicorn-web stdout | 2023-01-17 06:19:49,067 [231] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:19:49,535 [231] [DEBUG] [urllib3.connectionpool] https://api.bitbucket.org:443 "POST /2.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/hooks HTTP/1.1" 201 1140 gunicorn-web stdout | 2023-01-17 06:19:49,537 [231] [DEBUG] [peewee] ('UPDATE "repositorybuildtrigger" SET "uuid" = %s, "service_id" = %s, "repository_id" = %s, "connected_user_id" = %s, "secure_auth_token" = %s, "secure_private_key" = %s, "fully_migrated" = %s, "config" = %s, "write_token_id" = %s, "pull_robot_id" = %s, "enabled" = %s, "disabled_reason_id" = %s, "disabled_datetime" = %s, "successive_failure_count" = %s, "successive_internal_error_count" = %s WHERE ("repositorybuildtrigger"."id" = %s)', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 3, 1, 2, 'v0$$DNUePH52YwR2/Hmuv7Lbg9N+1H8lhYHL4SwJJMitBxFNKro+rMQVhDEpPDOgiedlmpx6+n+O5TKuqV3QuSxsXkWAqlIBb6W39wQH85P/nso=', 'v0$$MgaJQ8SBqlns7scEickVh/dGpbvrdurd9ewOjf/dAz2Z1VgJzcSebVXxU6Eq7roIQsUQfxO8TkX6coQ+ykh4gYu2GGKPZXywpObWv0rQF8ZGW7xnchKbbaFXmH2JaNxDqnLQ04bnom9+GSdY1cqRb0XAfQwR3rPXVqVR5qdxVaNwOy/OfjV67FyECb0VgwaJ4rSosEKcot2crTLH4vxO4pVxT04IxCnBcuh24j3qD1ha+nUlCifJMg2xfhcGMhOo7sh63IOsKtHPtBe2+H+jQY+F08mCFoThamBJ96oP0zaO9Yv2J1gdLYEG9NyJoDsOiJ37cAgZSWR85lTAW8K3UeHPldCJSL8N76bCrVA33o15QUdGkZjuCMIXDRRUDNh6JPm5akq2BJHKyJronxeKCnyxk/SO/2bj46ZlCimzdfoqq2SMfEkBxGT3fRmvPMEtFyGtjBCPJKoLNKxDxaiZDMW0KT6HJb2l8EptWp71InKR7KvzGQE2+VDm8t/od3TygwNwYBLgAiPDKMJPRu35RjgkFZyPN/XDBt18JLThRXUmHoHKO9R3SJiWyI8lutVbjTKQQi4ObgkJ4vSLTVbhGpITFx+9UfndQywwPprM7dYelJ9nH6087CaEFPWhIvWEsYOynA62jNhYmEIb0S1STyXPqrqPe6CoJKrtWFc6RHx4gBprSadlHxW453YTShAshkbh/zjPFYiJi9vVRzSmSYZT+UaN2NbdiCWGV1xUE64CttI7N4Bcfg9uAghDRii/fvJjwQD+qlP4S/KaDSnZvQzSPHn1hrW8uwN5Jo1B+Ny9BmqmjATBbUKK5F3ZJAJOQRfuPwSjAAQTZDKn4ZyBmgyhw3RIzOSKq5TTCFMadCyy5T6KFZz2MOXJWvChjcW289trneFNmycm0Sdo8qSpvViMdzSDf/T2UpHir72fICy9TCL4UZPTCw3A/4lxzOUBvlSE+gvFya14lO3/pUCJbpRcsfmRQcpe7wFjTaXa1XSzhEydpvK/cdobcx/pfjBR2m/CInHO54H8HvOSFtGgeV0RliFYXM+XMAWlrDwqbKVKn6lGSKlJjLCmmILRiO07QiASah+uTGdqEObz/o3zCH5TiPoB/qqOxdaQhKH3flDv7FZB0n8RByv9C3p9Wkgt+jmj4tub1y+ku7yWcysiCgxUC7N7YBWzCs4exCn96/K5Onyjqk3RykxYCZ7E/m5GbbzZ95XWt7W0OiJoSbZcGrK5qDickaXQ3cljYD4FcK9PhsoVw1YEXaOUtsyzdr6qJnLcGx82nYDxSMaap92yGTW35ovWEagQzkArBhOXkuLAJAIEn4JUKk/yIpndjG1kMk+Zii7Zmoty5STW9wkUvWBWYZZ3OuBhCNBLt0ob8x+Mo3jUQrH2bGvtbWuBQBtXSpITBy5KWpUcNFdK+LhDIfFwAWv8ZbRVPwpm+dwrW5bpVaVSK6/TmeGSzzcad3lLEKH9A0F7GqvByJhUH2HKywrD8B98viVpOowGBJdNjptN7ewz1EtvKgx+0DPYiv4GvO5PrSlmdmjT4ZUt0BRjxf0GLaS1TrM/OEEpxYXeK4eOgOZ3+wYtSfHrTeablKWr33q3Uo5zcyRRk62ci65c9sTTnB/jN8m3SIXgaNQcDeDLCo5Zu45D7fHhE0lYuBezlDGYvbsyMvvLn5YiY4M4/uUDABgLjKTxXKzK11CMLlWIbnClmYhBdeccDR+2sU7AM3tYmH3WefYHzkZA+IkkGw77Pt5HD4O3mGUgCqUC84y0L6YL95aGRHkvhPLj38KlKgJrUYxHAcvVzpENxhoSlTrwHvHQwodwo6O5EKFZP6UAx8vKaPjv6QwJVKahQSH25mvuskTHDjUkDTGRM70rUj0LbKy8ea7dIgjwsqKG6MOdCoJIVkdEeeBNjVJP7XW5Bi/PCw3Ecz3LN4aHzDwrdt8eC1w+6h+bR8V09Ucrcbp/9AL4kpZhDQ0vQaM5RetqoxvQHVRE6IZyUEhdH2LIwEOmcUiqY1iMlFKNSjozoU/qMKp11q1oIWOS0JbN0zKWevTmCBp8Mozl8jS3AZZTeBNNVZl6ir7huO/wP6EvGSPwxZ6ZVSrkskX91i+Y8QgZEkmkZOHnolNjW4FiCv6CkMk+W064zucAmaiOPQdNHlUm8cSyjzo/B5mVm53thBG8EPbcbD2E79LCjDIBtsRRhy9yTsPU2+LsJ106xim0CnmG+No2ZCzFzWTjKBlRsSoMsunOPw4EuALz896YhLaSrLFVXILsfdE/', False, '{"build_source": "{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test", "dockerfile_path": "//Dockerfile", "context": "/", "default_tag_from_ref": true, "latest_for_default_branch": true, "tag_templates": [], "credentials": [{"name": "SSH Public Key", "value": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGiUMaQPU6mKGhlyf8dPJliJ0qfN2zTWFFq15BzEt44RqlZCKmIqx2fe5AJba4QHRQzboT5SDBw3Zk5tRtyukpC+5JNZ1IMPgV2j1GH0wP9L0hDa1L1BIB8GEXu/afw7TQ/STyHCOoAMIjoZuOXJA/zsUtWs6s/fOTdrHQJndFViiR7miyOYOspAJ2pXsUlxu6sAkWZRyNv8QI8388/uwrSpWI4quNno6+ATmsnuAcM5xQAmwYnZeFGKbMoqHAt3Qq0Az4iGVpm/yn+utHKdPoIe6Flf4HD1pEtPQlIW7HzMxnOCpz01o6z17CfvhUoxusAmqbIGRPQL4gGR2NJnzF"}], "deploy_key_id": 7909538, "webhook_id": "{f6024ad8-7e49-46c8-85f7-4ebcb594816b}"}', 1, None, True, None, datetime.datetime(2023, 1, 17, 6, 19, 11, 521115), 0, 0, 1]) securityworker stdout | 2023-01-17 06:19:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:19:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:16.936988+00:00 (in 27.399046 seconds) securityworker stdout | 2023-01-17 06:19:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:19 UTC)" (scheduled at 2023-01-17 06:19:49.537472+00:00) securityworker stdout | 2023-01-17 06:19:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:19:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:19:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:19:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:19:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:19:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:19:49,545 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:49,549 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:49,552 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "logentrykind" AS "t1"', []) securityworker stdout | 2023-01-17 06:19:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:19:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:19:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:19:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:19:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:19:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:19:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:19:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:19 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:49,557 [231] [DEBUG] [peewee] ('INSERT INTO "logentry3" ("kind_id", "account_id", "performer_id", "repository_id", "datetime", "ip", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "logentry3"."id"', [53, 3, 2, 1, datetime.datetime(2023, 1, 17, 6, 19, 49, 549016), '10.131.0.6', '{"repo": "bitbucket", "namespace": "build_test", "trigger_id": "03697f15-5e57-4d5a-ae56-cb0fcc4b4f44", "service": "bitbucket", "pull_robot": null, "config": {"build_source": "{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test", "dockerfile_path": "//Dockerfile", "context": "/", "default_tag_from_ref": true, "latest_for_default_branch": true, "tag_templates": [], "credentials": [{"name": "SSH Public Key", "value": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGiUMaQPU6mKGhlyf8dPJliJ0qfN2zTWFFq15BzEt44RqlZCKmIqx2fe5AJba4QHRQzboT5SDBw3Zk5tRtyukpC+5JNZ1IMPgV2j1GH0wP9L0hDa1L1BIB8GEXu/afw7TQ/STyHCOoAMIjoZuOXJA/zsUtWs6s/fOTdrHQJndFViiR7miyOYOspAJ2pXsUlxu6sAkWZRyNv8QI8388/uwrSpWI4quNno6+ATmsnuAcM5xQAmwYnZeFGKbMoqHAt3Qq0Az4iGVpm/yn+utHKdPoIe6Flf4HD1pEtPQlIW7HzMxnOCpz01o6z17CfvhUoxusAmqbIGRPQL4gGR2NJnzF"}], "deploy_key_id": 7909538, "webhook_id": "{f6024ad8-7e49-46c8-85f7-4ebcb594816b}"}}']) gunicorn-web stdout | 2023-01-17 06:19:49,562 [231] [DEBUG] [app] Ending request: urn:request:cda4f162-ba57-477c-b51c-a5ce9327e183 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/activate) {'endpoint': 'api.buildtriggeractivate', 'request_id': 'urn:request:cda4f162-ba57-477c-b51c-a5ce9327e183', 'remote_addr': '10.131.0.6', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/activate', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/activate', 'parameters': {}, 'json_body': {'config': {'build_source': '{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test', 'dockerfile_path': '//Dockerfile', 'context': '/', 'default_tag_from_ref': True, 'latest_for_default_branch': True, 'tag_templates': [], 'credentials': [{'name': 'SSH Public Key', 'value': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGiUMaQPU6mKGhlyf8dPJliJ0qfN2zTWFFq15BzEt44RqlZCKmIqx2fe5AJba4QHRQzboT5SDBw3Zk5tRtyukpC+5JNZ1IMPgV2j1GH0wP9L0hDa1L1BIB8GEXu/afw7TQ/STyHCOoAMIjoZuOXJA/zsUtWs6s/fOTdrHQJndFViiR7miyOYOspAJ2pXsUlxu6sAkWZRyNv8QI8388/uwrSpWI4quNno6+ATmsnuAcM5xQAmwYnZeFGKbMoqHAt3Qq0Az4iGVpm/yn+utHKdPoIe6Flf4HD1pEtPQlIW7HzMxnOCpz01o6z17CfvhUoxusAmqbIGRPQL4gGR2NJnzF'}], 'deploy_key_id': 7909538, 'webhook_id': '{f6024ad8-7e49-46c8-85f7-4ebcb594816b}'}}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:49,563 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:49,563 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:19:49 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/activate HTTP/1.0" 200 1067 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:19:49 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/activate HTTP/2.0" 200 1067 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (2.049 2059 2.049) builder stdout | 2023-01-17 06:19:50,346 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:19:50,359 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:19:50,359 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:19:50,369 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:19:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:19:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:21.348605+00:00 (in 29.999502 seconds) gcworker stdout | 2023-01-17 06:19:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:21 UTC)" (scheduled at 2023-01-17 06:19:51.348605+00:00) gcworker stdout | 2023-01-17 06:19:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726791349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:19:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:19:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:19:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:52,060 [236] [DEBUG] [app] Starting request: urn:request:42fd30cd-4306-43ac-b268-ec79834e9f3d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:52,060 [231] [DEBUG] [app] Starting request: urn:request:273fb222-315d-44df-98af-91cfd8945ff0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:19:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:19:52,064 [230] [DEBUG] [app] Starting request: urn:request:f88555df-c5cc-4d56-96dc-fd30cb9dc6d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:52,065 [243] [DEBUG] [app] Starting request: urn:request:8ddd004d-b80c-488a-a145-dd04556ba95a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:19:52,065 [230] [DEBUG] [app] Ending request: urn:request:f88555df-c5cc-4d56-96dc-fd30cb9dc6d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f88555df-c5cc-4d56-96dc-fd30cb9dc6d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:19:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:19:52,065 [243] [DEBUG] [app] Ending request: urn:request:8ddd004d-b80c-488a-a145-dd04556ba95a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8ddd004d-b80c-488a-a145-dd04556ba95a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:19:52,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:19:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:52,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:19:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:19:52,070 [235] [DEBUG] [app] Starting request: urn:request:70eaaa21-4f0c-4272-8d1b-957739a3dcfd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:52,070 [235] [DEBUG] [app] Ending request: urn:request:70eaaa21-4f0c-4272-8d1b-957739a3dcfd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70eaaa21-4f0c-4272-8d1b-957739a3dcfd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:52,070 [233] [DEBUG] [app] Starting request: urn:request:ae31ea98-140e-4917-8459-663a9630a34a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:19:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:19:52,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:52,070 [233] [DEBUG] [app] Ending request: urn:request:ae31ea98-140e-4917-8459-663a9630a34a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae31ea98-140e-4917-8459-663a9630a34a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:19:52,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:19:52,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:19:52,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:19:52,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:52,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:19:52,087 [236] [DEBUG] [app] Ending request: urn:request:42fd30cd-4306-43ac-b268-ec79834e9f3d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42fd30cd-4306-43ac-b268-ec79834e9f3d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:52,087 [231] [DEBUG] [app] Ending request: urn:request:273fb222-315d-44df-98af-91cfd8945ff0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:273fb222-315d-44df-98af-91cfd8945ff0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:19:52,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:52,087 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) gunicorn-web stdout | 2023-01-17 06:19:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:19:52,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:19:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:19:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:19:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:01.636362+00:00 (in 5.989478 seconds) notificationworker stdout | 2023-01-17 06:19:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:05 UTC)" (scheduled at 2023-01-17 06:19:55.646354+00:00) notificationworker stdout | 2023-01-17 06:19:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:19:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 19, 55, 647376), True, datetime.datetime(2023, 1, 17, 6, 19, 55, 647376), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:19:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:19:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:19:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:05 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:19:58,690 [231] [DEBUG] [app] Starting request: urn:request:88e0ff66-fee2-4642-b3c5-1f864b0d02b9 (/api/v1/repository/build_test/bitbucket/tag/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:19:58,690 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,690 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,690 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:58,692 [236] [DEBUG] [app] Starting request: urn:request:79816372-429a-4628-9a29-de7847dd55a8 (/api/v1/repository/build_test/bitbucket) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:19:58,692 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,692 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,693 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:58,701 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:58,701 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:58,701 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,702 [231] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:58,702 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,702 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,702 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,702 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,703 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:58,703 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:58,703 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,704 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:58,704 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,704 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:58,704 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,704 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,704 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,707 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:58,711 [231] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,711 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:58,714 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:58,714 [236] [DEBUG] [endpoints.api.repository] Get repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:58,715 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:58,716 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:58,719 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."user_id", "t1"."repository_id", "t1"."created" FROM "star" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."user_id" = %s)) LIMIT %s OFFSET %s', [1, 2, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:58,720 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id", "t2"."id", "t2"."digest", "t2"."media_type_id", "t2"."layers_compressed_size", "t2"."config_media_type" FROM "tag" AS "t1" INNER JOIN "manifest" AS "t2" ON ("t1"."manifest_id" = "t2"."id") WHERE ((("t1"."repository_id" = %s) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" > %s))) AND ("t1"."hidden" = %s)) ORDER BY "t1"."lifetime_start_ms" DESC, "t1"."name" LIMIT %s OFFSET %s', [1, None, 1673936398720, False, 101, 0]) gunicorn-web stdout | 2023-01-17 06:19:58,723 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."count", "t1"."date" FROM "repositoryactioncount" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."date" >= %s))', [1, datetime.date(2022, 10, 17)]) gunicorn-web stdout | 2023-01-17 06:19:58,724 [231] [DEBUG] [app] Ending request: urn:request:88e0ff66-fee2-4642-b3c5-1f864b0d02b9 (/api/v1/repository/build_test/bitbucket/tag/) {'endpoint': 'api.listrepositorytags', 'request_id': 'urn:request:88e0ff66-fee2-4642-b3c5-1f864b0d02b9', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/tag/?limit=100&page=1&onlyActiveTags=true', 'path': '/api/v1/repository/build_test/bitbucket/tag/', 'parameters': {'limit': '100', 'page': '1', 'onlyActiveTags': 'true'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:58,725 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:19:58 +0000] "GET /api/v1/repository/build_test/bitbucket/tag/?limit=100&page=1&onlyActiveTags=true HTTP/2.0" 200 49 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.037 1713 0.037) gunicorn-web stdout | 2023-01-17 06:19:58,725 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:19:58 +0000] "GET /api/v1/repository/build_test/bitbucket/tag/?limit=100&page=1&onlyActiveTags=true HTTP/1.0" 200 49 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:19:58,726 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,726 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:58,728 [236] [DEBUG] [app] Ending request: urn:request:79816372-429a-4628-9a29-de7847dd55a8 (/api/v1/repository/build_test/bitbucket) {'endpoint': 'api.repository', 'request_id': 'urn:request:79816372-429a-4628-9a29-de7847dd55a8', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket?includeStats=true&includeTags=false', 'path': '/api/v1/repository/build_test/bitbucket', 'parameters': {'includeStats': 'true', 'includeTags': 'false'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:58,729 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:19:58 +0000] "GET /api/v1/repository/build_test/bitbucket?includeStats=true&includeTags=false HTTP/2.0" 200 3629 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.039 89 0.039) gunicorn-web stdout | 2023-01-17 06:19:58,729 [236] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:19:58 +0000] "GET /api/v1/repository/build_test/bitbucket?includeStats=true&includeTags=false HTTP/1.0" 200 3629 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" gunicorn-web stdout | 2023-01-17 06:19:59,348 [235] [DEBUG] [app] Starting request: urn:request:3ece3fed-c793-412c-86f8-dfb41b819f63 (/api/v1/repository/build_test/bitbucket/trigger/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:19:59,348 [235] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,348 [235] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,349 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,359 [235] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,361 [235] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:59,368 [235] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,369 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,374 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") WHERE (("t4"."username" = %s) AND ("t3"."name" = %s))', ['build_test', 'bitbucket']) gunicorn-web stdout | 2023-01-17 06:19:59,378 [235] [DEBUG] [app] Ending request: urn:request:3ece3fed-c793-412c-86f8-dfb41b819f63 (/api/v1/repository/build_test/bitbucket/trigger/) {'endpoint': 'api.buildtriggerlist', 'request_id': 'urn:request:3ece3fed-c793-412c-86f8-dfb41b819f63', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/', 'path': '/api/v1/repository/build_test/bitbucket/trigger/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:59,379 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:59,379 [235] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:19:59 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/1.0" 200 1083 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:19:59 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/2.0" 200 1083 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.033 415 0.033) gunicorn-web stdout | 2023-01-17 06:19:59,748 [231] [DEBUG] [app] Starting request: urn:request:4aca20bd-90b8-4584-9a4b-5896bcee74e9 (/repository/build_test/bitbucket/status) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:19:59,749 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,756 [236] [DEBUG] [app] Starting request: urn:request:f18378ff-e8d6-4963-bc3f-37e82a964f9d (/api/v1/repository/build_test/bitbucket/trigger/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:19:59,756 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,756 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,757 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,760 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,763 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "visibility" AS "t3" ON ("t1"."visibility_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t3"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 'public', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,766 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,768 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:19:59,769 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s OFFSET %s', ['bitbucket', 'build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,775 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:19:59,775 [231] [DEBUG] [app] Ending request: urn:request:4aca20bd-90b8-4584-9a4b-5896bcee74e9 (/repository/build_test/bitbucket/status) {'endpoint': 'web.build_status_badge', 'request_id': 'urn:request:4aca20bd-90b8-4584-9a4b-5896bcee74e9', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket/status?token=b4e9a60a-75cd-472a-9193-30aa274a686e', 'path': '/repository/build_test/bitbucket/status', 'parameters': {'token': 'b4e9a60a-75cd-472a-9193-30aa274a686e'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:59,776 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:19:59,776 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:59,776 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:19:59 +0000] "GET /repository/build_test/bitbucket/status?token=b4e9a60a-75cd-472a-9193-30aa274a686e HTTP/1.0" 200 740 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:19:59 +0000] "GET /repository/build_test/bitbucket/status?token=b4e9a60a-75cd-472a-9193-30aa274a686e HTTP/2.0" 200 387 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.030 154 0.030) gunicorn-web stdout | 2023-01-17 06:19:59,780 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") WHERE (("t4"."username" = %s) AND ("t3"."name" = %s))', ['build_test', 'bitbucket']) gunicorn-web stdout | 2023-01-17 06:19:59,785 [236] [DEBUG] [app] Ending request: urn:request:f18378ff-e8d6-4963-bc3f-37e82a964f9d (/api/v1/repository/build_test/bitbucket/trigger/) {'endpoint': 'api.buildtriggerlist', 'request_id': 'urn:request:f18378ff-e8d6-4963-bc3f-37e82a964f9d', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/', 'path': '/api/v1/repository/build_test/bitbucket/trigger/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:19:59,785 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:19:59,786 [236] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:19:59 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/1.0" 200 1083 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:19:59 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/2.0" 200 1083 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.032 69 0.031) gunicorn-web stdout | 2023-01-17 06:19:59,990 [231] [DEBUG] [app] Starting request: urn:request:aa9c73a9-2f63-4179-87f6-8254fbb3a084 (/api/v1/repository/build_test/bitbucket/build/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:19:59,990 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,990 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:19:59,991 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:20:00,000 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:20:00,001 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:20:00,001 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:20:00,001 [231] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:20:00,001 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:20:00,001 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,001 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,001 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,003 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:20:00,009 [231] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,010 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:20:00,016 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s', ['bitbucket', 'build_test', 3]) gunicorn-web stdout | 2023-01-17 06:20:00,021 [231] [DEBUG] [app] Ending request: urn:request:aa9c73a9-2f63-4179-87f6-8254fbb3a084 (/api/v1/repository/build_test/bitbucket/build/) {'endpoint': 'api.repositorybuildlist', 'request_id': 'urn:request:aa9c73a9-2f63-4179-87f6-8254fbb3a084', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/build/?limit=3', 'path': '/api/v1/repository/build_test/bitbucket/build/', 'parameters': {'limit': '3'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:20:00,021 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:00,022 [231] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:20:00 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:20:00 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.032 422 0.032) gunicorn-web stdout | 2023-01-17 06:20:00,322 [236] [DEBUG] [app] Starting request: urn:request:d6aef043-4a49-4dd9-a464-c193fe11b116 (/api/v1/repository/build_test/bitbucket/build/) {'X-Forwarded-For': '10.131.0.6'} gunicorn-web stdout | 2023-01-17 06:20:00,322 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:20:00,322 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:20:00,323 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,333 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,335 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:20:00,342 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:20:00,343 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:20:00,348 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s', ['bitbucket', 'build_test', 10]) gunicorn-web stdout | 2023-01-17 06:20:00,353 [236] [DEBUG] [app] Ending request: urn:request:d6aef043-4a49-4dd9-a464-c193fe11b116 (/api/v1/repository/build_test/bitbucket/build/) {'endpoint': 'api.repositorybuildlist', 'request_id': 'urn:request:d6aef043-4a49-4dd9-a464-c193fe11b116', 'remote_addr': '10.131.0.6', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/build/?limit=10', 'path': '/api/v1/repository/build_test/bitbucket/build/', 'parameters': {'limit': '10'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:20:00,354 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:00,354 [236] [INFO] [gunicorn.access] 10.131.0.6 - - [17/Jan/2023:06:20:00 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=10 HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.131.0.6 (-) - - [17/Jan/2023:06:20:00 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=10 HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.034 1674 0.034) builder stdout | 2023-01-17 06:20:00,375 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:00,386 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:00,386 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:00,397 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:01,375 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:20:01,415 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:20:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:05.646354+00:00 (in 4.009556 seconds) notificationworker stdout | 2023-01-17 06:20:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:01 UTC)" (scheduled at 2023-01-17 06:20:01.636362+00:00) notificationworker stdout | 2023-01-17 06:20:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:20:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:20:01,758 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:20:01,766 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:20:01,768 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:20:01,771 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:20:01,773 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} queuecleanupworker stdout | 2023-01-17 06:20:02,350 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:20:02,764 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:20:02,846 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:20:02,880 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:20:03,321 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:20:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:20:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:19.346120+00:00 (in 14.408842 seconds) repositorygcworker stdout | 2023-01-17 06:20:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:04 UTC)" (scheduled at 2023-01-17 06:20:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:20:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:20:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 4, 937725), True, datetime.datetime(2023, 1, 17, 6, 20, 4, 937725), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:20:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:20:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:20:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:20:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:15.646354+00:00 (in 9.999449 seconds) notificationworker stdout | 2023-01-17 06:20:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:15 UTC)" (scheduled at 2023-01-17 06:20:05.646354+00:00) notificationworker stdout | 2023-01-17 06:20:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 5, 647355), True, datetime.datetime(2023, 1, 17, 6, 20, 5, 647355), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:20:05,735 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:20:05,738 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:20:05,740 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:20:06,235 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:20:06,244 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:20:06,247 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:20:06,257 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:20:06,260 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:20:06,263 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:20:06,266 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:20:06,269 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:20:06,272 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:20:06,496 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:20:06,621 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:20:07,059 [231] [DEBUG] [app] Starting request: urn:request:f27f0720-afb0-4734-a938-78adb3970f61 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:07,059 [236] [DEBUG] [app] Starting request: urn:request:697dfd0d-7c76-4e30-aa18-ff36d93dba4f (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:07,064 [230] [DEBUG] [app] Starting request: urn:request:3ce75a75-0d3c-43ed-a691-a6d4de64810b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:07,064 [242] [DEBUG] [app] Starting request: urn:request:f4a9fd84-b398-4359-8080-b3cf7f0e3739 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:07,065 [230] [DEBUG] [app] Ending request: urn:request:3ce75a75-0d3c-43ed-a691-a6d4de64810b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ce75a75-0d3c-43ed-a691-a6d4de64810b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:20:07,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:20:07,065 [242] [DEBUG] [app] Ending request: urn:request:f4a9fd84-b398-4359-8080-b3cf7f0e3739 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4a9fd84-b398-4359-8080-b3cf7f0e3739', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:07,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:20:07,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:20:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:07,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:07,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:07,069 [235] [DEBUG] [app] Starting request: urn:request:e0acac5e-ed9d-4cb1-b542-8285b0f04fdb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:07,069 [233] [DEBUG] [app] Starting request: urn:request:8b5f4234-2894-4725-9acd-5148624c89d2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:07,069 [235] [DEBUG] [app] Ending request: urn:request:e0acac5e-ed9d-4cb1-b542-8285b0f04fdb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0acac5e-ed9d-4cb1-b542-8285b0f04fdb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:07,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:07,070 [233] [DEBUG] [app] Ending request: urn:request:8b5f4234-2894-4725-9acd-5148624c89d2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b5f4234-2894-4725-9acd-5148624c89d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:07,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:07,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:20:07,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:07,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:07,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:07,086 [236] [DEBUG] [app] Ending request: urn:request:697dfd0d-7c76-4e30-aa18-ff36d93dba4f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:697dfd0d-7c76-4e30-aa18-ff36d93dba4f', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:07,086 [231] [DEBUG] [app] Ending request: urn:request:f27f0720-afb0-4734-a938-78adb3970f61 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f27f0720-afb0-4734-a938-78adb3970f61', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:07,086 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:07,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:07,086 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:20:07,086 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) storagereplication stdout | 2023-01-17 06:20:07,758 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:20:07,767 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:20:08,202 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:20:08,605 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:20:08,623 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:20:08,841 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:20:09,086 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:20:09,542 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:20:10,101 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:20:10,237 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 06:20:10,407 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:10,420 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:10,420 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:10,431 [60] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 06:20:10,500 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:20:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:20:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:40.738568+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 06:20:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:40 UTC)" (scheduled at 2023-01-17 06:20:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:20:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 20, 10, 739442), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:20:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:20:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:20:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:20:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:25.646354+00:00 (in 9.999558 seconds) notificationworker stdout | 2023-01-17 06:20:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:25 UTC)" (scheduled at 2023-01-17 06:20:15.646354+00:00) notificationworker stdout | 2023-01-17 06:20:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 15, 647237), True, datetime.datetime(2023, 1, 17, 6, 20, 15, 647237), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:20:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:19.537472+00:00 (in 2.599992 seconds) securityworker stdout | 2023-01-17 06:20:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:46 UTC)" (scheduled at 2023-01-17 06:20:16.936988+00:00) securityworker stdout | 2023-01-17 06:20:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:20:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:20:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:04.936835+00:00 (in 45.590255 seconds) repositorygcworker stdout | 2023-01-17 06:20:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:19 UTC)" (scheduled at 2023-01-17 06:20:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:20:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:20:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:20:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:46.936988+00:00 (in 27.399029 seconds) securityworker stdout | 2023-01-17 06:20:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:49 UTC)" (scheduled at 2023-01-17 06:20:19.537472+00:00) securityworker stdout | 2023-01-17 06:20:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:20:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:20:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:20:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:20:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:20:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:20:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:49 UTC)" executed successfully builder stdout | 2023-01-17 06:20:20,437 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:20,449 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:20,449 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:20,460 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:21,349 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:20:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:51.348605+00:00 (in 29.999438 seconds) gcworker stdout | 2023-01-17 06:20:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:51 UTC)" (scheduled at 2023-01-17 06:20:21.348605+00:00) gcworker stdout | 2023-01-17 06:20:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726821349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:20:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:20:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:20:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:20:22,060 [231] [DEBUG] [app] Starting request: urn:request:e893a1d9-f4cc-40c9-b5ad-7cea1a7270a9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:22,060 [236] [DEBUG] [app] Starting request: urn:request:dfd588da-d516-40a0-a7d4-e915ed20644d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:22,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:22,065 [230] [DEBUG] [app] Starting request: urn:request:aa7708ef-c775-4dbd-b1f1-5dfee99b888c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:22,065 [243] [DEBUG] [app] Starting request: urn:request:bed8cdac-e79f-43b3-bd29-25322fbf38ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:22,066 [230] [DEBUG] [app] Ending request: urn:request:aa7708ef-c775-4dbd-b1f1-5dfee99b888c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa7708ef-c775-4dbd-b1f1-5dfee99b888c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:20:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:20:22,066 [243] [DEBUG] [app] Ending request: urn:request:bed8cdac-e79f-43b3-bd29-25322fbf38ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bed8cdac-e79f-43b3-bd29-25322fbf38ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:20:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:20:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:22,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:22,071 [235] [DEBUG] [app] Starting request: urn:request:416a3f83-8fa9-4287-87c3-10b4731f1e5f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:22,071 [231] [DEBUG] [app] Starting request: urn:request:2960f8b2-b86c-4a35-a106-5b4e7852946e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:22,071 [235] [DEBUG] [app] Ending request: urn:request:416a3f83-8fa9-4287-87c3-10b4731f1e5f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:416a3f83-8fa9-4287-87c3-10b4731f1e5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:22,071 [231] [DEBUG] [app] Ending request: urn:request:2960f8b2-b86c-4a35-a106-5b4e7852946e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2960f8b2-b86c-4a35-a106-5b4e7852946e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:20:22,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:22,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:22,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:22,088 [231] [DEBUG] [app] Ending request: urn:request:e893a1d9-f4cc-40c9-b5ad-7cea1a7270a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e893a1d9-f4cc-40c9-b5ad-7cea1a7270a9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:22,088 [236] [DEBUG] [app] Ending request: urn:request:dfd588da-d516-40a0-a7d4-e915ed20644d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dfd588da-d516-40a0-a7d4-e915ed20644d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:20:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:20:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:25,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:35.646354+00:00 (in 9.999469 seconds) notificationworker stdout | 2023-01-17 06:20:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:35 UTC)" (scheduled at 2023-01-17 06:20:25.646354+00:00) notificationworker stdout | 2023-01-17 06:20:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 25, 647421), True, datetime.datetime(2023, 1, 17, 6, 20, 25, 647421), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:20:28,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:20:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:35.853396+00:00 (in 6.889303 seconds) exportactionlogsworker stdout | 2023-01-17 06:20:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:28 UTC)" (scheduled at 2023-01-17 06:20:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:20:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:20:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:28 UTC)" executed successfully builder stdout | 2023-01-17 06:20:30,470 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:30,483 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:30,483 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:30,493 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:31,412 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:20:31,434 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:20:31,768 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:20:31,776 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:20:31,780 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:20:31,783 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:20:31,787 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} queuecleanupworker stdout | 2023-01-17 06:20:32,385 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:20:32,791 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:20:32,870 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:20:32,898 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:20:33,340 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:20:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:35,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:45.646354+00:00 (in 9.999407 seconds) notificationworker stdout | 2023-01-17 06:20:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:45 UTC)" (scheduled at 2023-01-17 06:20:35.646354+00:00) notificationworker stdout | 2023-01-17 06:20:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 35, 647386), True, datetime.datetime(2023, 1, 17, 6, 20, 35, 647386), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:35,660 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:35,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:35,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:20:35,744 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:20:35,747 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:20:35,749 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:20:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:20:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:28.963662+00:00 (in 53.109766 seconds) exportactionlogsworker stdout | 2023-01-17 06:20:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:35 UTC)" (scheduled at 2023-01-17 06:20:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:20:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:20:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 35, 854344), True, datetime.datetime(2023, 1, 17, 6, 20, 35, 854344), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:20:35,868 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:20:35,868 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:20:35,869 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:20:36,242 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:20:36,254 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:20:36,257 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:20:36,267 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:20:36,270 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:20:36,272 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:20:36,275 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:20:36,278 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:20:36,280 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:20:36,502 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:20:36,631 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:20:37,060 [236] [DEBUG] [app] Starting request: urn:request:c7f1e469-6a1d-4d94-a504-5a98625adab5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:37,060 [231] [DEBUG] [app] Starting request: urn:request:79681a76-9db5-44c7-bdf5-f96a7e93dec0 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:37,065 [243] [DEBUG] [app] Starting request: urn:request:5cfb320c-b5f9-46f1-be63-5535b8c92be3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:37,065 [230] [DEBUG] [app] Starting request: urn:request:7c8e5866-4b6d-4a58-ae87-03c4d3bfbcea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:37,065 [243] [DEBUG] [app] Ending request: urn:request:5cfb320c-b5f9-46f1-be63-5535b8c92be3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cfb320c-b5f9-46f1-be63-5535b8c92be3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:20:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:20:37,066 [230] [DEBUG] [app] Ending request: urn:request:7c8e5866-4b6d-4a58-ae87-03c4d3bfbcea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7c8e5866-4b6d-4a58-ae87-03c4d3bfbcea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:20:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:20:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:37,070 [235] [DEBUG] [app] Starting request: urn:request:36876f5d-4cb1-49ef-9910-081519806e44 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:37,070 [235] [DEBUG] [app] Ending request: urn:request:36876f5d-4cb1-49ef-9910-081519806e44 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36876f5d-4cb1-49ef-9910-081519806e44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:37,070 [233] [DEBUG] [app] Starting request: urn:request:d1e1534b-bf9b-443f-a1ac-f220da8732ef (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:20:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:37,071 [233] [DEBUG] [app] Ending request: urn:request:d1e1534b-bf9b-443f-a1ac-f220da8732ef (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d1e1534b-bf9b-443f-a1ac-f220da8732ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:20:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:37,088 [236] [DEBUG] [app] Ending request: urn:request:c7f1e469-6a1d-4d94-a504-5a98625adab5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c7f1e469-6a1d-4d94-a504-5a98625adab5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:37,088 [231] [DEBUG] [app] Ending request: urn:request:79681a76-9db5-44c7-bdf5-f96a7e93dec0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79681a76-9db5-44c7-bdf5-f96a7e93dec0', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:37,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:37,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:37,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:20:37,089 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 06:20:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:20:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:37.537407+00:00 (in 0.280346 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:20:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:37 UTC)" (scheduled at 2023-01-17 06:20:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:20:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:20:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 37, 257560), True, datetime.datetime(2023, 1, 17, 6, 20, 37, 257560), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:20:37,270 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:20:37,270 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:20:37,270 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:20:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:20:37,538 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:23.537048+00:00 (in 45.999016 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:20:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:37 UTC)" (scheduled at 2023-01-17 06:20:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:20:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:20:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:37 UTC)" executed successfully servicekey stdout | 2023-01-17 06:20:37,777 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} storagereplication stdout | 2023-01-17 06:20:37,792 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} repositorygcworker stdout | 2023-01-17 06:20:38,238 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:20:38,620 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:20:38,644 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:20:38,871 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:20:39,102 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:20:39,549 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:20:40,137 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:20:40,273 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} builder stdout | 2023-01-17 06:20:40,494 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:40,507 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:40,507 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:40,518 [60] [DEBUG] [data.database] Disconnecting from database. blobuploadcleanupworker stdout | 2023-01-17 06:20:40,535 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:20:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:20:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:10.738568+00:00 (in 29.999537 seconds) buildlogsarchiver stdout | 2023-01-17 06:20:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:10 UTC)" (scheduled at 2023-01-17 06:20:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:20:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 20, 40, 739455), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:20:40,752 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:20:40,753 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:20:40,753 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:20:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:20:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:42.941536+00:00 (in 1.792797 seconds) namespacegcworker stdout | 2023-01-17 06:20:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:41 UTC)" (scheduled at 2023-01-17 06:20:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:20:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:20:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 41, 149186), True, datetime.datetime(2023, 1, 17, 6, 20, 41, 149186), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:20:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:20:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:20:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:20:42,942 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:20:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:41.148245+00:00 (in 58.206110 seconds) namespacegcworker stdout | 2023-01-17 06:20:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:42 UTC)" (scheduled at 2023-01-17 06:20:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:20:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:20:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:21:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:20:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:55.646354+00:00 (in 9.999565 seconds) notificationworker stdout | 2023-01-17 06:20:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:55 UTC)" (scheduled at 2023-01-17 06:20:45.646354+00:00) notificationworker stdout | 2023-01-17 06:20:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 45, 647195), True, datetime.datetime(2023, 1, 17, 6, 20, 45, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:45,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:20:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:20:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:20:49.537472+00:00 (in 2.600026 seconds) securityworker stdout | 2023-01-17 06:20:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:16 UTC)" (scheduled at 2023-01-17 06:20:46.936988+00:00) securityworker stdout | 2023-01-17 06:20:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:20:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:20:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:16.936988+00:00 (in 27.399090 seconds) securityworker stdout | 2023-01-17 06:20:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:19 UTC)" (scheduled at 2023-01-17 06:20:49.537472+00:00) securityworker stdout | 2023-01-17 06:20:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:20:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:20:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:20:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:20:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:20:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:20:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:20:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:20:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:20:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:20:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:20:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:20:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:20:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:19 UTC)" executed successfully builder stdout | 2023-01-17 06:20:50,528 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:20:50,540 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:20:50,540 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:20:50,551 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:20:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:21.348605+00:00 (in 29.999534 seconds) gcworker stdout | 2023-01-17 06:20:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:21 UTC)" (scheduled at 2023-01-17 06:20:51.348605+00:00) gcworker stdout | 2023-01-17 06:20:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726851349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:20:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:20:51,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:20:51,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:20:52,059 [236] [DEBUG] [app] Starting request: urn:request:4602d024-596e-4269-b3d5-6c82b90a9a4c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:52,060 [231] [DEBUG] [app] Starting request: urn:request:3279efc6-2158-4a43-9f25-b9e242359bd4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:20:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:20:52,064 [243] [DEBUG] [app] Starting request: urn:request:259e2cd6-0185-4f5e-9a3e-65d4bb90d206 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:52,065 [230] [DEBUG] [app] Starting request: urn:request:1a297492-7b52-4da5-9564-db0124dd5f08 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:20:52,065 [243] [DEBUG] [app] Ending request: urn:request:259e2cd6-0185-4f5e-9a3e-65d4bb90d206 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:259e2cd6-0185-4f5e-9a3e-65d4bb90d206', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:20:52,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:20:52,065 [230] [DEBUG] [app] Ending request: urn:request:1a297492-7b52-4da5-9564-db0124dd5f08 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1a297492-7b52-4da5-9564-db0124dd5f08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:52,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:20:52,065 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:20:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:52,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:20:52,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:20:52,069 [235] [DEBUG] [app] Starting request: urn:request:354701de-82fb-43b7-a13c-375abb08eff2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:52,069 [236] [DEBUG] [app] Starting request: urn:request:e175058f-5ec3-4ba2-ba03-7e9c356fbbaf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:20:52,070 [235] [DEBUG] [app] Ending request: urn:request:354701de-82fb-43b7-a13c-375abb08eff2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:354701de-82fb-43b7-a13c-375abb08eff2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:20:52,070 [236] [DEBUG] [app] Ending request: urn:request:e175058f-5ec3-4ba2-ba03-7e9c356fbbaf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e175058f-5ec3-4ba2-ba03-7e9c356fbbaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:20:52,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:52,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:20:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:20:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:52,070 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:20:52,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:52,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:20:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:52,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:20:52,086 [236] [DEBUG] [app] Ending request: urn:request:4602d024-596e-4269-b3d5-6c82b90a9a4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4602d024-596e-4269-b3d5-6c82b90a9a4c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:52,086 [231] [DEBUG] [app] Ending request: urn:request:3279efc6-2158-4a43-9f25-b9e242359bd4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3279efc6-2158-4a43-9f25-b9e242359bd4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:20:52,086 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:52,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:20:52,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:20:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:20:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) notificationworker stdout | 2023-01-17 06:20:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:20:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:01.636362+00:00 (in 5.989576 seconds) notificationworker stdout | 2023-01-17 06:20:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:05 UTC)" (scheduled at 2023-01-17 06:20:55.646354+00:00) notificationworker stdout | 2023-01-17 06:20:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:20:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 20, 55, 647204), True, datetime.datetime(2023, 1, 17, 6, 20, 55, 647204), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:20:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:20:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:20:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:05 UTC)" executed successfully builder stdout | 2023-01-17 06:21:00,561 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:00,573 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:00,574 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:00,584 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:01,431 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:21:01,468 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:21:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:01,637 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:05.646354+00:00 (in 4.009526 seconds) notificationworker stdout | 2023-01-17 06:21:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:01 UTC)" (scheduled at 2023-01-17 06:21:01.636362+00:00) notificationworker stdout | 2023-01-17 06:21:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:21:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:21:01,777 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:21:01,786 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:21:01,788 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:21:01,791 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:21:01,796 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} queuecleanupworker stdout | 2023-01-17 06:21:02,421 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:21:02,827 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:21:02,892 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:21:02,932 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:21:03,376 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:21:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:21:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:17.345601+00:00 (in 12.408321 seconds) repositorygcworker stdout | 2023-01-17 06:21:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:04 UTC)" (scheduled at 2023-01-17 06:21:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:21:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:21:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 4, 937709), True, datetime.datetime(2023, 1, 17, 6, 21, 4, 937709), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:21:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:21:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:21:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:21:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:15.646354+00:00 (in 9.999464 seconds) notificationworker stdout | 2023-01-17 06:21:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:15 UTC)" (scheduled at 2023-01-17 06:21:05.646354+00:00) notificationworker stdout | 2023-01-17 06:21:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 5, 647342), True, datetime.datetime(2023, 1, 17, 6, 21, 5, 647342), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:21:05,751 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:21:05,754 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:21:05,756 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:21:06,253 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:21:06,263 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:21:06,265 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:21:06,275 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:21:06,278 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:21:06,281 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:21:06,284 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:21:06,286 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:21:06,288 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:21:06,513 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:21:06,650 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:21:07,060 [231] [DEBUG] [app] Starting request: urn:request:2e805b52-6bd2-4108-8dc4-abf0f1a0b046 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:07,060 [236] [DEBUG] [app] Starting request: urn:request:a7d92cf6-562f-4d5a-bddb-6d70c94d37b4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:07,065 [243] [DEBUG] [app] Starting request: urn:request:d826de04-f167-42d8-8e4c-d372042dc970 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:07,065 [230] [DEBUG] [app] Starting request: urn:request:d5d40e00-4a07-4cb8-b2ce-1e3bbc734c2d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:07,065 [243] [DEBUG] [app] Ending request: urn:request:d826de04-f167-42d8-8e4c-d372042dc970 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d826de04-f167-42d8-8e4c-d372042dc970', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-registry stdout | 2023-01-17 06:21:07,066 [230] [DEBUG] [app] Ending request: urn:request:d5d40e00-4a07-4cb8-b2ce-1e3bbc734c2d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d5d40e00-4a07-4cb8-b2ce-1e3bbc734c2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:21:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:21:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:07,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:07,070 [235] [DEBUG] [app] Starting request: urn:request:25607c46-c06e-49db-b30f-0d3d84632f85 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:07,070 [236] [DEBUG] [app] Starting request: urn:request:3c6cad71-0b46-484c-ba4e-6a808c600058 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:07,070 [235] [DEBUG] [app] Ending request: urn:request:25607c46-c06e-49db-b30f-0d3d84632f85 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25607c46-c06e-49db-b30f-0d3d84632f85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:07,070 [236] [DEBUG] [app] Ending request: urn:request:3c6cad71-0b46-484c-ba4e-6a808c600058 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c6cad71-0b46-484c-ba4e-6a808c600058', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:07,070 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:07,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:07,087 [236] [DEBUG] [app] Ending request: urn:request:a7d92cf6-562f-4d5a-bddb-6d70c94d37b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7d92cf6-562f-4d5a-bddb-6d70c94d37b4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:07,087 [231] [DEBUG] [app] Ending request: urn:request:2e805b52-6bd2-4108-8dc4-abf0f1a0b046 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e805b52-6bd2-4108-8dc4-abf0f1a0b046', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:21:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" storagereplication stdout | 2023-01-17 06:21:07,802 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:21:07,812 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:21:08,274 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:21:08,656 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:21:08,679 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:21:08,902 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:21:09,137 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:21:09,585 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:21:10,173 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:21:10,308 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:21:10,541 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 06:21:10,594 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:10,607 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:10,607 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:10,617 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:21:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:21:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:40.738568+00:00 (in 29.999528 seconds) buildlogsarchiver stdout | 2023-01-17 06:21:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:40 UTC)" (scheduled at 2023-01-17 06:21:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:21:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 21, 10, 739441), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:21:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:21:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:21:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:40 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:21:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:25.646354+00:00 (in 9.999540 seconds) notificationworker stdout | 2023-01-17 06:21:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:25 UTC)" (scheduled at 2023-01-17 06:21:15.646354+00:00) notificationworker stdout | 2023-01-17 06:21:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:15,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 15, 647227), True, datetime.datetime(2023, 1, 17, 6, 21, 15, 647227), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:15,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:15,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:15,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:21:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:19.537472+00:00 (in 2.599991 seconds) securityworker stdout | 2023-01-17 06:21:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:46 UTC)" (scheduled at 2023-01-17 06:21:16.936988+00:00) securityworker stdout | 2023-01-17 06:21:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:21:17,345 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:21:17,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:19.346120+00:00 (in 2.000020 seconds) repositorygcworker stdout | 2023-01-17 06:21:17,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:17 UTC)" (scheduled at 2023-01-17 06:21:17.345601+00:00) repositorygcworker stdout | 2023-01-17 06:21:17,347 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 21, 17, 346530), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:21:17,357 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 17, 346530), True, datetime.datetime(2023, 1, 17, 6, 21, 17, 346530), 0, 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:21:17,361 [80] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 17, 346530), True, datetime.datetime(2023, 1, 17, 6, 21, 17, 346530), 0, 'repositorygc/%', False, datetime.datetime(2023, 1, 17, 6, 21, 17, 346530), 'repositorygc/%']) repositorygcworker stdout | 2023-01-17 06:21:17,364 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:21:17,364 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:17 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:21:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:21:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:04.936835+00:00 (in 45.590242 seconds) repositorygcworker stdout | 2023-01-17 06:21:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:19 UTC)" (scheduled at 2023-01-17 06:21:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:21:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:21:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:21:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:46.936988+00:00 (in 27.399039 seconds) securityworker stdout | 2023-01-17 06:21:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:49 UTC)" (scheduled at 2023-01-17 06:21:19.537472+00:00) securityworker stdout | 2023-01-17 06:21:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:21:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:19,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:21:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:19,546 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:19,557 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:19,558 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:21:19,558 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:21:19,558 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:21:19,558 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:19,559 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:19,559 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:19,559 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:21:19,559 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:49 UTC)" executed successfully builder stdout | 2023-01-17 06:21:20,627 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:20,640 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:20,640 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:20,651 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:21:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:51.348605+00:00 (in 29.999569 seconds) gcworker stdout | 2023-01-17 06:21:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:51 UTC)" (scheduled at 2023-01-17 06:21:21.348605+00:00) gcworker stdout | 2023-01-17 06:21:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726881349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:21:21,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:21:21,361 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:21,361 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:21:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:21:22,059 [236] [DEBUG] [app] Starting request: urn:request:68a9e621-52ec-4b72-b637-4652af458c3c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:22,059 [231] [DEBUG] [app] Starting request: urn:request:029d462c-8d25-4261-af8d-b43750dc19ca (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:22,060 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,062 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:22,064 [243] [DEBUG] [app] Starting request: urn:request:a4f3896b-7d0b-4ad7-ad56-e3f414f30438 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:22,064 [241] [DEBUG] [app] Starting request: urn:request:4e86070d-8986-4241-8bd0-a793ea9a6dd9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:22,064 [243] [DEBUG] [app] Ending request: urn:request:a4f3896b-7d0b-4ad7-ad56-e3f414f30438 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4f3896b-7d0b-4ad7-ad56-e3f414f30438', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:21:22,065 [241] [DEBUG] [app] Ending request: urn:request:4e86070d-8986-4241-8bd0-a793ea9a6dd9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e86070d-8986-4241-8bd0-a793ea9a6dd9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:21:22,065 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:21:22,065 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-web stdout | 2023-01-17 06:21:22,066 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,066 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:22,068 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:22,068 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:22,069 [235] [DEBUG] [app] Starting request: urn:request:9ba29939-b417-416e-8a8b-5f7b21f48ffa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:22,069 [235] [DEBUG] [app] Ending request: urn:request:9ba29939-b417-416e-8a8b-5f7b21f48ffa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ba29939-b417-416e-8a8b-5f7b21f48ffa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:22,069 [233] [DEBUG] [app] Starting request: urn:request:10201a82-db0d-4930-b193-5cdd208e0de4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:22,069 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:22,069 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:22,070 [233] [DEBUG] [app] Ending request: urn:request:10201a82-db0d-4930-b193-5cdd208e0de4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10201a82-db0d-4930-b193-5cdd208e0de4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:22,070 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:22,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:22,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:22,070 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:22,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:22,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:22,086 [231] [DEBUG] [app] Ending request: urn:request:029d462c-8d25-4261-af8d-b43750dc19ca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:029d462c-8d25-4261-af8d-b43750dc19ca', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:22,086 [236] [DEBUG] [app] Ending request: urn:request:68a9e621-52ec-4b72-b637-4652af458c3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:68a9e621-52ec-4b72-b637-4652af458c3c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:22,086 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:22,086 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:22,086 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:21:22,086 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) securityscanningnotificationworker stdout | 2023-01-17 06:21:23,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:21:23,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:37.256652+00:00 (in 13.719006 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:21:23,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:23 UTC)" (scheduled at 2023-01-17 06:21:23.537048+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:21:23,538 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 21, 23, 537978), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:21:23,548 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 23, 537978), True, datetime.datetime(2023, 1, 17, 6, 21, 23, 537978), 0, 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:21:23,552 [81] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 23, 537978), True, datetime.datetime(2023, 1, 17, 6, 21, 23, 537978), 0, 'secscanv4/%', False, datetime.datetime(2023, 1, 17, 6, 21, 23, 537978), 'secscanv4/%']) securityscanningnotificationworker stdout | 2023-01-17 06:21:23,555 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:21:23,556 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:23 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:21:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:35.646354+00:00 (in 9.999552 seconds) notificationworker stdout | 2023-01-17 06:21:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:35 UTC)" (scheduled at 2023-01-17 06:21:25.646354+00:00) notificationworker stdout | 2023-01-17 06:21:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:25,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 25, 647198), True, datetime.datetime(2023, 1, 17, 6, 21, 25, 647198), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:25,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:25,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:21:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:21:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:35.853396+00:00 (in 6.889241 seconds) exportactionlogsworker stdout | 2023-01-17 06:21:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:28 UTC)" (scheduled at 2023-01-17 06:21:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:21:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:21:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:28 UTC)" executed successfully builder stdout | 2023-01-17 06:21:30,661 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:30,673 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:30,673 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:30,683 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:31,440 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:21:31,503 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:21:31,785 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:21:31,795 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:21:31,798 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:21:31,801 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:21:31,804 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} queuecleanupworker stdout | 2023-01-17 06:21:32,457 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} expiredappspecifictokenworker stdout | 2023-01-17 06:21:32,746 [64] [DEBUG] [apscheduler.scheduler] Looking for jobs to run expiredappspecifictokenworker stdout | 2023-01-17 06:21:32,746 [64] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 07:21:32.744591+00:00 (in 3599.998075 seconds) expiredappspecifictokenworker stdout | 2023-01-17 06:21:32,746 [64] [INFO] [apscheduler.executors.default] Running job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 07:21:32 UTC)" (scheduled at 2023-01-17 06:21:32.744591+00:00) expiredappspecifictokenworker stdout | 2023-01-17 06:21:32,746 [64] [DEBUG] [__main__] Garbage collecting expired app specific tokens with window: 1 day, 0:00:00 expiredappspecifictokenworker stdout | 2023-01-17 06:21:32,747 [64] [DEBUG] [peewee] ('DELETE FROM "appspecificauthtoken" WHERE ("appspecificauthtoken"."expiration" < %s)', [datetime.datetime(2023, 1, 16, 6, 21, 32, 746969)]) expiredappspecifictokenworker stdout | 2023-01-17 06:21:32,756 [64] [DEBUG] [data.database] Disconnecting from database. expiredappspecifictokenworker stdout | 2023-01-17 06:21:32,757 [64] [INFO] [apscheduler.executors.default] Job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2023-01-17 07:21:32 UTC)" executed successfully logrotateworker stdout | 2023-01-17 06:21:32,857 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:21:32,927 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:21:32,940 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:21:33,393 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:21:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:45.646354+00:00 (in 9.999583 seconds) notificationworker stdout | 2023-01-17 06:21:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:45 UTC)" (scheduled at 2023-01-17 06:21:35.646354+00:00) notificationworker stdout | 2023-01-17 06:21:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 35, 647195), True, datetime.datetime(2023, 1, 17, 6, 21, 35, 647195), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:21:35,759 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:21:35,761 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:21:35,764 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:21:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:21:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:54.963413+00:00 (in 19.109514 seconds) exportactionlogsworker stdout | 2023-01-17 06:21:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:35 UTC)" (scheduled at 2023-01-17 06:21:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:21:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:21:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 35, 854303), True, datetime.datetime(2023, 1, 17, 6, 21, 35, 854303), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:21:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:21:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:21:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:21:36,260 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:21:36,272 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:21:36,274 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:21:36,282 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:21:36,285 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:21:36,287 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:21:36,290 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:21:36,294 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:21:36,296 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:21:36,549 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:21:36,683 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:21:37,059 [235] [DEBUG] [app] Starting request: urn:request:25a7537d-d711-4391-a373-cbbf724fa25d (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:37,060 [236] [DEBUG] [app] Starting request: urn:request:2a94a2f8-33ca-4fe9-b9d2-0d6bf2682a53 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:37,064 [243] [DEBUG] [app] Starting request: urn:request:6e0f7d51-8382-46e8-b088-2707b0160755 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:37,065 [242] [DEBUG] [app] Starting request: urn:request:2fb44861-6085-4d23-b930-3b98c0a7b6f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:37,065 [243] [DEBUG] [app] Ending request: urn:request:6e0f7d51-8382-46e8-b088-2707b0160755 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e0f7d51-8382-46e8-b088-2707b0160755', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:37,065 [242] [DEBUG] [app] Ending request: urn:request:2fb44861-6085-4d23-b930-3b98c0a7b6f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2fb44861-6085-4d23-b930-3b98c0a7b6f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:37,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:21:37,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:37,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:21:37,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:37,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:37,070 [231] [DEBUG] [app] Starting request: urn:request:bdd68233-3c68-49cd-b4c6-1c927635cf76 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:37,070 [233] [DEBUG] [app] Starting request: urn:request:1bc8f62d-b054-45d1-bd47-9995023a7d25 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:37,070 [231] [DEBUG] [app] Ending request: urn:request:bdd68233-3c68-49cd-b4c6-1c927635cf76 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bdd68233-3c68-49cd-b4c6-1c927635cf76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:37,071 [233] [DEBUG] [app] Ending request: urn:request:1bc8f62d-b054-45d1-bd47-9995023a7d25 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1bc8f62d-b054-45d1-bd47-9995023a7d25', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:37,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:37,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:37,087 [236] [DEBUG] [app] Ending request: urn:request:2a94a2f8-33ca-4fe9-b9d2-0d6bf2682a53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a94a2f8-33ca-4fe9-b9d2-0d6bf2682a53', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:37,088 [235] [DEBUG] [app] Ending request: urn:request:25a7537d-d711-4391-a373-cbbf724fa25d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:25a7537d-d711-4391-a373-cbbf724fa25d', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:21:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.030) securityscanningnotificationworker stdout | 2023-01-17 06:21:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:21:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:37.537407+00:00 (in 0.280317 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:21:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:37 UTC)" (scheduled at 2023-01-17 06:21:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:21:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:21:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 37, 257559), True, datetime.datetime(2023, 1, 17, 6, 21, 37, 257559), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:21:37,269 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:21:37,269 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:21:37,269 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:21:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:21:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:37.256652+00:00 (in 59.718836 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:21:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:37 UTC)" (scheduled at 2023-01-17 06:21:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:21:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:21:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:21:37,808 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:21:37,835 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:21:38,312 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:21:38,692 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:21:38,694 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:21:38,934 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:21:39,151 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:21:39,601 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:21:40,209 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:21:40,344 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:21:40,575 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} builder stdout | 2023-01-17 06:21:40,692 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:40,705 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:40,705 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:40,715 [60] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:21:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:21:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:10.738568+00:00 (in 29.999562 seconds) buildlogsarchiver stdout | 2023-01-17 06:21:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:10 UTC)" (scheduled at 2023-01-17 06:21:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:21:40,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 21, 40, 739384), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:21:40,749 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:21:40,749 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:21:40,749 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:10 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:21:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:21:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:42.941536+00:00 (in 1.792702 seconds) namespacegcworker stdout | 2023-01-17 06:21:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:41 UTC)" (scheduled at 2023-01-17 06:21:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:21:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:21:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 41, 149245), True, datetime.datetime(2023, 1, 17, 6, 21, 41, 149245), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:21:41,161 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:21:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:21:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:21:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:21:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:06.941185+00:00 (in 23.999189 seconds) namespacegcworker stdout | 2023-01-17 06:21:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:42 UTC)" (scheduled at 2023-01-17 06:21:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:21:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:21:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:22:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:21:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:55.646354+00:00 (in 9.999549 seconds) notificationworker stdout | 2023-01-17 06:21:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:55 UTC)" (scheduled at 2023-01-17 06:21:45.646354+00:00) notificationworker stdout | 2023-01-17 06:21:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 45, 647253), True, datetime.datetime(2023, 1, 17, 6, 21, 45, 647253), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:45,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:21:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:21:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:21:49.537472+00:00 (in 2.600018 seconds) securityworker stdout | 2023-01-17 06:21:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:16 UTC)" (scheduled at 2023-01-17 06:21:46.936988+00:00) securityworker stdout | 2023-01-17 06:21:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:46,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:21:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:21:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:16.936988+00:00 (in 27.399084 seconds) securityworker stdout | 2023-01-17 06:21:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:19 UTC)" (scheduled at 2023-01-17 06:21:49.537472+00:00) securityworker stdout | 2023-01-17 06:21:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:21:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:21:49,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:21:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:21:49,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:21:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:21:49,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:49,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:21:49,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:21:49,554 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:21:49,554 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:21:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:21:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:21:49,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:21:49,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:19 UTC)" executed successfully builder stdout | 2023-01-17 06:21:50,726 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:21:50,738 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:21:50,738 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:21:50,749 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:21:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:21.348605+00:00 (in 29.999539 seconds) gcworker stdout | 2023-01-17 06:21:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:21 UTC)" (scheduled at 2023-01-17 06:21:51.348605+00:00) gcworker stdout | 2023-01-17 06:21:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726911349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:21:51,361 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:21:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:21:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:21:52,059 [235] [DEBUG] [app] Starting request: urn:request:21ab3c71-e00e-41bb-b7ed-9ac951929e83 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:52,059 [231] [DEBUG] [app] Starting request: urn:request:acbafbef-fa37-4ac6-a9a7-571762ba5c33 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:21:52,060 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,060 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:52,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:21:52,064 [242] [DEBUG] [app] Starting request: urn:request:4fd02b4e-11c5-49a6-a169-2a77eb1b3d47 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:52,064 [242] [DEBUG] [app] Ending request: urn:request:4fd02b4e-11c5-49a6-a169-2a77eb1b3d47 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4fd02b4e-11c5-49a6-a169-2a77eb1b3d47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:21:52,064 [240] [DEBUG] [app] Starting request: urn:request:bfaf11ad-25d1-4ef9-adfe-8af47391992c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:21:52,065 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:21:52,065 [240] [DEBUG] [app] Ending request: urn:request:bfaf11ad-25d1-4ef9-adfe-8af47391992c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bfaf11ad-25d1-4ef9-adfe-8af47391992c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:52,065 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:21:52,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:21:52,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:52,066 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:21:52,068 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:52,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:21:52,069 [236] [DEBUG] [app] Starting request: urn:request:9be795ba-cc36-45f1-b14d-148bbadf15a2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:52,069 [233] [DEBUG] [app] Starting request: urn:request:a2ed94af-325b-45de-9f09-6597c5979df5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:21:52,070 [236] [DEBUG] [app] Ending request: urn:request:9be795ba-cc36-45f1-b14d-148bbadf15a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9be795ba-cc36-45f1-b14d-148bbadf15a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:52,070 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:52,070 [233] [DEBUG] [app] Ending request: urn:request:a2ed94af-325b-45de-9f09-6597c5979df5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a2ed94af-325b-45de-9f09-6597c5979df5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:21:52,070 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:21:52,070 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:21:52,070 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:21:52,070 [235] [INFO] [data.database] Connection pooling disabled for postgresql nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:21:52,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:21:52,077 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:52,077 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:21:52,086 [231] [DEBUG] [app] Ending request: urn:request:acbafbef-fa37-4ac6-a9a7-571762ba5c33 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:acbafbef-fa37-4ac6-a9a7-571762ba5c33', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:52,086 [235] [DEBUG] [app] Ending request: urn:request:21ab3c71-e00e-41bb-b7ed-9ac951929e83 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21ab3c71-e00e-41bb-b7ed-9ac951929e83', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:21:52,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:21:52,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:21:52,087 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:21:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) exportactionlogsworker stdout | 2023-01-17 06:21:54,963 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:21:54,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:28.963662+00:00 (in 33.999793 seconds) exportactionlogsworker stdout | 2023-01-17 06:21:54,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:54 UTC)" (scheduled at 2023-01-17 06:21:54.963413+00:00) exportactionlogsworker stdout | 2023-01-17 06:21:54,964 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 21, 54, 964276), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:21:54,975 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 54, 964276), True, datetime.datetime(2023, 1, 17, 6, 21, 54, 964276), 0, 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:21:54,979 [65] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 21, 54, 964276), True, datetime.datetime(2023, 1, 17, 6, 21, 54, 964276), 0, 'exportactionlogs/%', False, datetime.datetime(2023, 1, 17, 6, 21, 54, 964276), 'exportactionlogs/%']) exportactionlogsworker stdout | 2023-01-17 06:21:54,983 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:21:54,983 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:26:54 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:21:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:21:55,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:01.636362+00:00 (in 5.989518 seconds) notificationworker stdout | 2023-01-17 06:21:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:05 UTC)" (scheduled at 2023-01-17 06:21:55.646354+00:00) notificationworker stdout | 2023-01-17 06:21:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:21:55,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 21, 55, 647239), True, datetime.datetime(2023, 1, 17, 6, 21, 55, 647239), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:21:55,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:21:55,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:21:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:05 UTC)" executed successfully builder stdout | 2023-01-17 06:22:00,758 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:00,770 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:00,770 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:00,781 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:01,475 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:22:01,538 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:22:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:05.646354+00:00 (in 4.009552 seconds) notificationworker stdout | 2023-01-17 06:22:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:01 UTC)" (scheduled at 2023-01-17 06:22:01.636362+00:00) notificationworker stdout | 2023-01-17 06:22:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:22:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:01,794 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:22:01,804 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:22:01,806 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:22:01,809 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:22:01,812 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} queuecleanupworker stdout | 2023-01-17 06:22:02,492 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:22:02,892 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:22:02,944 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:22:02,975 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:22:03,417 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:22:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:22:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:19.346120+00:00 (in 14.408875 seconds) repositorygcworker stdout | 2023-01-17 06:22:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:04 UTC)" (scheduled at 2023-01-17 06:22:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:22:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:22:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 4, 937657), True, datetime.datetime(2023, 1, 17, 6, 22, 4, 937657), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:22:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:22:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:22:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:22:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:05,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:15.646354+00:00 (in 9.999505 seconds) notificationworker stdout | 2023-01-17 06:22:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:15 UTC)" (scheduled at 2023-01-17 06:22:05.646354+00:00) notificationworker stdout | 2023-01-17 06:22:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:05,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 5, 647243), True, datetime.datetime(2023, 1, 17, 6, 22, 5, 647243), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:05,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:05,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:05,658 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:22:05,766 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:22:05,769 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:22:05,771 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:22:06,270 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:22:06,280 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:22:06,283 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:22:06,288 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:22:06,291 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:22:06,295 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:22:06,298 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:22:06,300 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:22:06,306 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:22:06,585 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:22:06,704 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} namespacegcworker stdout | 2023-01-17 06:22:06,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:22:06,941 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:41.148245+00:00 (in 34.206570 seconds) namespacegcworker stdout | 2023-01-17 06:22:06,941 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:27:06 UTC)" (scheduled at 2023-01-17 06:22:06.941185+00:00) namespacegcworker stdout | 2023-01-17 06:22:06,942 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 22, 6, 942067), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:22:06,953 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 22, 6, 942067), True, datetime.datetime(2023, 1, 17, 6, 22, 6, 942067), 0, 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:22:06,957 [74] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 22, 6, 942067), True, datetime.datetime(2023, 1, 17, 6, 22, 6, 942067), 0, 'namespacegc/%', False, datetime.datetime(2023, 1, 17, 6, 22, 6, 942067), 'namespacegc/%']) namespacegcworker stdout | 2023-01-17 06:22:06,960 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:22:06,960 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:27:06 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:07,060 [231] [DEBUG] [app] Starting request: urn:request:d315416a-8204-4058-b415-aaa683f906b9 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:07,060 [236] [DEBUG] [app] Starting request: urn:request:b24a9067-e83f-439c-8877-e39c83f44089 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:07,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:07,064 [240] [DEBUG] [app] Starting request: urn:request:a3bb0993-deea-4970-8317-30410821b969 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:07,065 [240] [DEBUG] [app] Ending request: urn:request:a3bb0993-deea-4970-8317-30410821b969 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3bb0993-deea-4970-8317-30410821b969', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:07,065 [241] [DEBUG] [app] Starting request: urn:request:7f7b3848-5ffe-48e5-a751-9f0c1cb84b63 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:07,065 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:22:07,065 [241] [DEBUG] [app] Ending request: urn:request:7f7b3848-5ffe-48e5-a751-9f0c1cb84b63 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f7b3848-5ffe-48e5-a751-9f0c1cb84b63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:07,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:22:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:22:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:07,070 [235] [DEBUG] [app] Starting request: urn:request:be2d2cce-2cbe-4a4d-9296-2b0897ee9778 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:07,070 [231] [DEBUG] [app] Starting request: urn:request:e2a9c7a7-e01e-4dd3-8f82-6f481cfc4cc0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:07,071 [235] [DEBUG] [app] Ending request: urn:request:be2d2cce-2cbe-4a4d-9296-2b0897ee9778 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be2d2cce-2cbe-4a4d-9296-2b0897ee9778', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:07,071 [231] [DEBUG] [app] Ending request: urn:request:e2a9c7a7-e01e-4dd3-8f82-6f481cfc4cc0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e2a9c7a7-e01e-4dd3-8f82-6f481cfc4cc0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:22:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:07,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:07,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:07,077 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:07,086 [236] [DEBUG] [app] Ending request: urn:request:b24a9067-e83f-439c-8877-e39c83f44089 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b24a9067-e83f-439c-8877-e39c83f44089', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:07,087 [231] [DEBUG] [app] Ending request: urn:request:d315416a-8204-4058-b415-aaa683f906b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d315416a-8204-4058-b415-aaa683f906b9', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:07,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:07,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.028) gunicorn-web stdout | 2023-01-17 06:22:07,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) storagereplication stdout | 2023-01-17 06:22:07,831 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:22:07,869 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:22:08,343 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:22:08,718 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:22:08,729 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:22:08,970 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:22:09,182 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:22:09,621 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:22:10,240 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:22:10,354 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:22:10,609 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:22:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:22:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:40.738568+00:00 (in 29.999527 seconds) buildlogsarchiver stdout | 2023-01-17 06:22:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:40 UTC)" (scheduled at 2023-01-17 06:22:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:22:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 22, 10, 739455), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:22:10,750 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:22:10,750 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:22:10,750 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:40 UTC)" executed successfully builder stdout | 2023-01-17 06:22:10,791 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:10,803 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:10,803 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:10,813 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:25.646354+00:00 (in 9.999532 seconds) notificationworker stdout | 2023-01-17 06:22:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:25 UTC)" (scheduled at 2023-01-17 06:22:15.646354+00:00) notificationworker stdout | 2023-01-17 06:22:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 15, 647188), True, datetime.datetime(2023, 1, 17, 6, 22, 15, 647188), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:22:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:19.537472+00:00 (in 2.600035 seconds) securityworker stdout | 2023-01-17 06:22:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:46 UTC)" (scheduled at 2023-01-17 06:22:16.936988+00:00) securityworker stdout | 2023-01-17 06:22:16,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:16,937 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:16,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:16,941 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:16,951 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:16,954 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:22:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:22:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:04.936835+00:00 (in 45.590224 seconds) repositorygcworker stdout | 2023-01-17 06:22:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:19 UTC)" (scheduled at 2023-01-17 06:22:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:22:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:22:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:22:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:46.936988+00:00 (in 27.399064 seconds) securityworker stdout | 2023-01-17 06:22:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:49 UTC)" (scheduled at 2023-01-17 06:22:19.537472+00:00) securityworker stdout | 2023-01-17 06:22:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:22:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:22:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:19,544 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:19,554 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:19,554 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:22:19,554 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:22:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:22:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:19,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:19,556 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:22:19,556 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:49 UTC)" executed successfully builder stdout | 2023-01-17 06:22:20,821 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:20,834 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:20,834 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:20,845 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:22:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:51.348605+00:00 (in 29.999511 seconds) gcworker stdout | 2023-01-17 06:22:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:51 UTC)" (scheduled at 2023-01-17 06:22:21.348605+00:00) gcworker stdout | 2023-01-17 06:22:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726941349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:22:21,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:22:21,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:21,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:22:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:22,060 [233] [DEBUG] [app] Starting request: urn:request:683ccd5d-7016-48fc-a813-dedf3a4a94b4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:22,060 [231] [DEBUG] [app] Starting request: urn:request:852fe8a5-5261-4165-9a2d-8e35b80d6ff4 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:22,065 [243] [DEBUG] [app] Starting request: urn:request:214a0510-6e35-45c4-9ba8-cfca29ff8c40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:22,065 [243] [DEBUG] [app] Ending request: urn:request:214a0510-6e35-45c4-9ba8-cfca29ff8c40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:214a0510-6e35-45c4-9ba8-cfca29ff8c40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:22,065 [242] [DEBUG] [app] Starting request: urn:request:e1b1831f-47d5-4b6b-8444-96f934bf12f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:22,065 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-registry stdout | 2023-01-17 06:22:22,065 [242] [DEBUG] [app] Ending request: urn:request:e1b1831f-47d5-4b6b-8444-96f934bf12f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e1b1831f-47d5-4b6b-8444-96f934bf12f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:22,065 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:22:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:22:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:22,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:22,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:22,070 [236] [DEBUG] [app] Starting request: urn:request:b948e3e9-41ba-4d75-88a1-3dd89f682fef (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:22,070 [236] [DEBUG] [app] Ending request: urn:request:b948e3e9-41ba-4d75-88a1-3dd89f682fef (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b948e3e9-41ba-4d75-88a1-3dd89f682fef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:22,070 [235] [DEBUG] [app] Starting request: urn:request:2ded416c-0840-467e-8fc5-2b6ea94deea4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:22:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:22,071 [235] [DEBUG] [app] Ending request: urn:request:2ded416c-0840-467e-8fc5-2b6ea94deea4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2ded416c-0840-467e-8fc5-2b6ea94deea4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:22,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:22:22,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:22,072 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:22,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:22,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:22,087 [233] [DEBUG] [app] Ending request: urn:request:683ccd5d-7016-48fc-a813-dedf3a4a94b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:683ccd5d-7016-48fc-a813-dedf3a4a94b4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:22,088 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:22,088 [231] [DEBUG] [app] Ending request: urn:request:852fe8a5-5261-4165-9a2d-8e35b80d6ff4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:852fe8a5-5261-4165-9a2d-8e35b80d6ff4', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:22,088 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.030 47 0.029) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) gunicorn-web stdout | 2023-01-17 06:22:22,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:22:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" notificationworker stdout | 2023-01-17 06:22:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:35.646354+00:00 (in 9.999531 seconds) notificationworker stdout | 2023-01-17 06:22:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:35 UTC)" (scheduled at 2023-01-17 06:22:25.646354+00:00) notificationworker stdout | 2023-01-17 06:22:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 25, 647294), True, datetime.datetime(2023, 1, 17, 6, 22, 25, 647294), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:22:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:22:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:35.853396+00:00 (in 6.889238 seconds) exportactionlogsworker stdout | 2023-01-17 06:22:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:28 UTC)" (scheduled at 2023-01-17 06:22:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:22:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:22:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:28 UTC)" executed successfully builder stdout | 2023-01-17 06:22:30,850 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:30,866 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:30,866 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:30,876 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:31,491 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:22:31,560 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:22:31,802 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:22:31,813 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:22:31,815 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:22:31,817 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:22:31,820 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} queuecleanupworker stdout | 2023-01-17 06:22:32,528 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:22:32,926 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:22:32,980 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:22:32,987 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:22:33,453 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} notificationworker stdout | 2023-01-17 06:22:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:45.646354+00:00 (in 9.999558 seconds) notificationworker stdout | 2023-01-17 06:22:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:45 UTC)" (scheduled at 2023-01-17 06:22:35.646354+00:00) notificationworker stdout | 2023-01-17 06:22:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:35,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 35, 647212), True, datetime.datetime(2023, 1, 17, 6, 22, 35, 647212), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:35,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:35,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:22:35,773 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:22:35,776 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:22:35,778 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:22:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:22:35,853 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:28.963662+00:00 (in 53.109812 seconds) exportactionlogsworker stdout | 2023-01-17 06:22:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:35 UTC)" (scheduled at 2023-01-17 06:22:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:22:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:22:35,855 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 35, 854270), True, datetime.datetime(2023, 1, 17, 6, 22, 35, 854270), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:22:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:22:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:22:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:35 UTC)" executed successfully gunicorn-registry stdout | 2023-01-17 06:22:36,281 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:22:36,287 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:22:36,291 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:22:36,297 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:22:36,299 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:22:36,302 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:22:36,304 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:22:36,311 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:22:36,315 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:22:36,616 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:22:36,729 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:22:37,060 [236] [DEBUG] [app] Starting request: urn:request:8ccbc0dc-415d-4421-9411-de088cef0417 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:37,060 [235] [DEBUG] [app] Starting request: urn:request:5ed54b78-2238-41ce-a1c7-c79db2b512d2 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,063 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:37,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:37,065 [230] [DEBUG] [app] Starting request: urn:request:71558e92-e29f-4b81-88e1-c5d25dfa15cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:37,066 [243] [DEBUG] [app] Starting request: urn:request:d615df2b-74ac-43ed-853d-a9e1e14b2f2e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:37,066 [230] [DEBUG] [app] Ending request: urn:request:71558e92-e29f-4b81-88e1-c5d25dfa15cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71558e92-e29f-4b81-88e1-c5d25dfa15cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:37,066 [243] [DEBUG] [app] Ending request: urn:request:d615df2b-74ac-43ed-853d-a9e1e14b2f2e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d615df2b-74ac-43ed-853d-a9e1e14b2f2e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:22:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:22:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:37,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:37,067 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) gunicorn-web stdout | 2023-01-17 06:22:37,068 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:37,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:37,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:37,070 [235] [DEBUG] [app] Starting request: urn:request:6bfd3550-21e9-454a-ade4-ebd1e8d60059 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:37,070 [231] [DEBUG] [app] Starting request: urn:request:104fc0f0-06d6-4cbe-8a8d-1de5cc28485e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:37,071 [235] [DEBUG] [app] Ending request: urn:request:6bfd3550-21e9-454a-ade4-ebd1e8d60059 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6bfd3550-21e9-454a-ade4-ebd1e8d60059', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:37,071 [231] [DEBUG] [app] Ending request: urn:request:104fc0f0-06d6-4cbe-8a8d-1de5cc28485e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:104fc0f0-06d6-4cbe-8a8d-1de5cc28485e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:37,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:22:37,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:37,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:37,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:37,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:37,079 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:37,088 [235] [DEBUG] [app] Ending request: urn:request:5ed54b78-2238-41ce-a1c7-c79db2b512d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ed54b78-2238-41ce-a1c7-c79db2b512d2', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:37,088 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:37,088 [236] [DEBUG] [app] Ending request: urn:request:8ccbc0dc-415d-4421-9411-de088cef0417 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ccbc0dc-415d-4421-9411-de088cef0417', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:37,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:22:37,088 [236] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:22:37,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) securityscanningnotificationworker stdout | 2023-01-17 06:22:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:22:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:37.537407+00:00 (in 0.280360 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:22:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:37 UTC)" (scheduled at 2023-01-17 06:22:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:22:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:22:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 37, 257458), True, datetime.datetime(2023, 1, 17, 6, 22, 37, 257458), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:22:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:22:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:22:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:37 UTC)" executed successfully securityscanningnotificationworker stdout | 2023-01-17 06:22:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:22:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:37.256652+00:00 (in 59.718890 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:22:37,537 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:37 UTC)" (scheduled at 2023-01-17 06:22:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:22:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:22:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:37 UTC)" executed successfully storagereplication stdout | 2023-01-17 06:22:37,868 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:22:37,887 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:22:38,350 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:22:38,728 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:22:38,764 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:22:39,006 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:22:39,217 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:22:39,650 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:22:40,253 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:22:40,389 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:22:40,644 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:22:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:22:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:10.738568+00:00 (in 29.999526 seconds) buildlogsarchiver stdout | 2023-01-17 06:22:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:10 UTC)" (scheduled at 2023-01-17 06:22:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:22:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 22, 40, 739448), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:22:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:22:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:22:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:10 UTC)" executed successfully builder stdout | 2023-01-17 06:22:40,884 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:40,896 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:40,896 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:40,907 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:22:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:22:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:42.941536+00:00 (in 1.792833 seconds) namespacegcworker stdout | 2023-01-17 06:22:41,148 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:41 UTC)" (scheduled at 2023-01-17 06:22:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:22:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:22:41,149 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 41, 149074), True, datetime.datetime(2023, 1, 17, 6, 22, 41, 149074), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:22:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:22:41,161 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:22:41,161 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:22:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:22:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:41.148245+00:00 (in 58.206249 seconds) namespacegcworker stdout | 2023-01-17 06:22:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:42 UTC)" (scheduled at 2023-01-17 06:22:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:22:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:22:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:23:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:22:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:55.646354+00:00 (in 9.999567 seconds) notificationworker stdout | 2023-01-17 06:22:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:55 UTC)" (scheduled at 2023-01-17 06:22:45.646354+00:00) notificationworker stdout | 2023-01-17 06:22:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:45,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 45, 647224), True, datetime.datetime(2023, 1, 17, 6, 22, 45, 647224), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:22:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:22:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:22:49.537472+00:00 (in 2.600014 seconds) securityworker stdout | 2023-01-17 06:22:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:16 UTC)" (scheduled at 2023-01-17 06:22:46.936988+00:00) securityworker stdout | 2023-01-17 06:22:46,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:46,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:16 UTC)" executed successfully securityworker stdout | 2023-01-17 06:22:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:22:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:16.936988+00:00 (in 27.399027 seconds) securityworker stdout | 2023-01-17 06:22:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:19 UTC)" (scheduled at 2023-01-17 06:22:49.537472+00:00) securityworker stdout | 2023-01-17 06:22:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:22:49,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:49,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:22:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:22:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:22:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:22:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:22:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:22:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:22:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:22:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:22:49,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:22:49,556 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:22:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:22:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:19 UTC)" executed successfully builder stdout | 2023-01-17 06:22:50,907 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:22:50,920 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:22:50,920 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:22:50,930 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:22:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:21.348605+00:00 (in 29.999520 seconds) gcworker stdout | 2023-01-17 06:22:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:21 UTC)" (scheduled at 2023-01-17 06:22:51.348605+00:00) gcworker stdout | 2023-01-17 06:22:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672726971349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:22:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:22:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:22:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:22:52,060 [231] [DEBUG] [app] Starting request: urn:request:c1ea1c19-4790-4004-8c1d-5bad74391bc8 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:52,060 [236] [DEBUG] [app] Starting request: urn:request:b38b0df5-1b94-4c91-8230-a1be75efda7c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:22:52,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:52,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:22:52,065 [242] [DEBUG] [app] Starting request: urn:request:5f8d99c8-8268-4ff9-ac02-90b017db15ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:52,065 [241] [DEBUG] [app] Starting request: urn:request:fc5de0f8-7f99-4198-93bd-7934c6194ab4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:22:52,065 [242] [DEBUG] [app] Ending request: urn:request:5f8d99c8-8268-4ff9-ac02-90b017db15ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f8d99c8-8268-4ff9-ac02-90b017db15ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:22:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:22:52,066 [241] [DEBUG] [app] Ending request: urn:request:fc5de0f8-7f99-4198-93bd-7934c6194ab4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc5de0f8-7f99-4198-93bd-7934c6194ab4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:52,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:22:52,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:52,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:22:52,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:22:52,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:52,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:22:52,071 [235] [DEBUG] [app] Starting request: urn:request:c7c83bc1-646f-45fd-b5f0-9397e7f63fdb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:52,071 [233] [DEBUG] [app] Starting request: urn:request:086464c7-0f5f-4762-8a97-7af69d2a63ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:22:52,071 [235] [DEBUG] [app] Ending request: urn:request:c7c83bc1-646f-45fd-b5f0-9397e7f63fdb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c7c83bc1-646f-45fd-b5f0-9397e7f63fdb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:52,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:52,071 [233] [DEBUG] [app] Ending request: urn:request:086464c7-0f5f-4762-8a97-7af69d2a63ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:086464c7-0f5f-4762-8a97-7af69d2a63ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:22:52,072 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:22:52,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:22:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:22:52,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:22:52,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:52,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:22:52,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:52,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:22:52,088 [236] [DEBUG] [app] Ending request: urn:request:b38b0df5-1b94-4c91-8230-a1be75efda7c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b38b0df5-1b94-4c91-8230-a1be75efda7c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:52,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:52,088 [231] [DEBUG] [app] Ending request: urn:request:c1ea1c19-4790-4004-8c1d-5bad74391bc8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c1ea1c19-4790-4004-8c1d-5bad74391bc8', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:22:52,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:22:52,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:22:52,089 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:22:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) notificationworker stdout | 2023-01-17 06:22:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:22:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:01.636362+00:00 (in 5.989548 seconds) notificationworker stdout | 2023-01-17 06:22:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:05 UTC)" (scheduled at 2023-01-17 06:22:55.646354+00:00) notificationworker stdout | 2023-01-17 06:22:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:22:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 22, 55, 647283), True, datetime.datetime(2023, 1, 17, 6, 22, 55, 647283), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:22:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:22:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:22:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:05 UTC)" executed successfully builder stdout | 2023-01-17 06:23:00,938 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:00,951 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:00,951 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:00,962 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:23:01,527 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:23:01,590 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:23:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:05.646354+00:00 (in 4.009583 seconds) notificationworker stdout | 2023-01-17 06:23:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:01 UTC)" (scheduled at 2023-01-17 06:23:01.636362+00:00) notificationworker stdout | 2023-01-17 06:23:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:23:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:01,811 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:23:01,820 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:23:01,823 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:23:01,825 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:23:01,828 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} queuecleanupworker stdout | 2023-01-17 06:23:02,564 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:23:02,941 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:23:02,985 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:23:03,022 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:23:03,465 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} repositorygcworker stdout | 2023-01-17 06:23:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:23:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:19.346120+00:00 (in 14.408798 seconds) repositorygcworker stdout | 2023-01-17 06:23:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:04 UTC)" (scheduled at 2023-01-17 06:23:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:23:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:23:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 4, 937707), True, datetime.datetime(2023, 1, 17, 6, 23, 4, 937707), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:23:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:23:04,949 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:23:04,949 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:23:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:15.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 06:23:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:15 UTC)" (scheduled at 2023-01-17 06:23:05.646354+00:00) notificationworker stdout | 2023-01-17 06:23:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 5, 647265), True, datetime.datetime(2023, 1, 17, 6, 23, 5, 647265), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:23:05,780 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:23:05,782 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:23:05,785 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:23:06,290 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:23:06,294 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:23:06,297 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:23:06,303 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:23:06,306 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:23:06,308 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:23:06,315 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:23:06,320 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:23:06,324 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} buildlogsarchiver stdout | 2023-01-17 06:23:06,640 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:23:06,760 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:23:07,060 [231] [DEBUG] [app] Starting request: urn:request:6e6091de-c214-4252-952d-33f12d424bf5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:07,060 [236] [DEBUG] [app] Starting request: urn:request:ae284ec9-4172-45a4-b3d9-d77f444924ee (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,062 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:07,065 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:07,065 [241] [DEBUG] [app] Starting request: urn:request:b32d0c0c-32c4-43a4-b5c3-a4edc453523d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:07,065 [241] [DEBUG] [app] Ending request: urn:request:b32d0c0c-32c4-43a4-b5c3-a4edc453523d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b32d0c0c-32c4-43a4-b5c3-a4edc453523d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:07,065 [242] [DEBUG] [app] Starting request: urn:request:f2083ef9-7964-4b6c-af53-3a2034c4a340 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:07,066 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:23:07,066 [242] [DEBUG] [app] Ending request: urn:request:f2083ef9-7964-4b6c-af53-3a2034c4a340 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f2083ef9-7964-4b6c-af53-3a2034c4a340', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:07,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:23:07,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,068 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:07,070 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:07,070 [235] [DEBUG] [app] Starting request: urn:request:73d6642a-aadb-4bca-8d5f-3bd4caf6a15b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:07,070 [233] [DEBUG] [app] Starting request: urn:request:6c1a1ab8-e6a8-42e5-a90b-636a46012eae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:07,070 [235] [DEBUG] [app] Ending request: urn:request:73d6642a-aadb-4bca-8d5f-3bd4caf6a15b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73d6642a-aadb-4bca-8d5f-3bd4caf6a15b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:07,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:07,071 [233] [DEBUG] [app] Ending request: urn:request:6c1a1ab8-e6a8-42e5-a90b-636a46012eae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6c1a1ab8-e6a8-42e5-a90b-636a46012eae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:07,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:07,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:23:07,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:07,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:07,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:07,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:07,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:07,087 [236] [DEBUG] [app] Ending request: urn:request:ae284ec9-4172-45a4-b3d9-d77f444924ee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ae284ec9-4172-45a4-b3d9-d77f444924ee', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:07,087 [231] [DEBUG] [app] Ending request: urn:request:6e6091de-c214-4252-952d-33f12d424bf5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e6091de-c214-4252-952d-33f12d424bf5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:07,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:07,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:07,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:23:07,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) storagereplication stdout | 2023-01-17 06:23:07,893 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:23:07,906 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} repositorygcworker stdout | 2023-01-17 06:23:08,372 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:23:08,763 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:23:08,799 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:23:09,037 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:23:09,244 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:23:09,686 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:23:10,288 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:23:10,422 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:23:10,654 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:23:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:23:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:40.738568+00:00 (in 29.999539 seconds) buildlogsarchiver stdout | 2023-01-17 06:23:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:40 UTC)" (scheduled at 2023-01-17 06:23:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:23:10,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 23, 10, 739440), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:23:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:23:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:23:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:40 UTC)" executed successfully builder stdout | 2023-01-17 06:23:10,972 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:10,984 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:10,984 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:10,995 [60] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:15,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:25.646354+00:00 (in 9.999555 seconds) notificationworker stdout | 2023-01-17 06:23:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:25 UTC)" (scheduled at 2023-01-17 06:23:15.646354+00:00) notificationworker stdout | 2023-01-17 06:23:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 15, 647214), True, datetime.datetime(2023, 1, 17, 6, 23, 15, 647214), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:15,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:15,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:15,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:25 UTC)" executed successfully securityworker stdout | 2023-01-17 06:23:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:19.537472+00:00 (in 2.599987 seconds) securityworker stdout | 2023-01-17 06:23:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:46 UTC)" (scheduled at 2023-01-17 06:23:16.936988+00:00) securityworker stdout | 2023-01-17 06:23:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:23:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:16,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:16,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:23:16,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:46 UTC)" executed successfully repositorygcworker stdout | 2023-01-17 06:23:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:23:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:04.936835+00:00 (in 45.590271 seconds) repositorygcworker stdout | 2023-01-17 06:23:19,346 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:19 UTC)" (scheduled at 2023-01-17 06:23:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:23:19,346 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:23:19,346 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:23:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:46.936988+00:00 (in 27.399019 seconds) securityworker stdout | 2023-01-17 06:23:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:49 UTC)" (scheduled at 2023-01-17 06:23:19.537472+00:00) securityworker stdout | 2023-01-17 06:23:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:23:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:23:19,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:23:19,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:23:19,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:23:19,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:23:19,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:19,556 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:23:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:23:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:49 UTC)" executed successfully builder stdout | 2023-01-17 06:23:20,998 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:21,010 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:21,010 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:21,021 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:23:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:23:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:51.348605+00:00 (in 29.999510 seconds) gcworker stdout | 2023-01-17 06:23:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:51 UTC)" (scheduled at 2023-01-17 06:23:21.348605+00:00) gcworker stdout | 2023-01-17 06:23:21,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672727001349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:23:21,372 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:23:21,372 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:23:21,372 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:23:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:22,060 [236] [DEBUG] [app] Starting request: urn:request:ead9331a-d61c-47e7-a884-55248b559ecc (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:22,060 [231] [DEBUG] [app] Starting request: urn:request:f566164a-8bed-46d1-822b-81f5bebdc02b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:22,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,063 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:22,064 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:22,065 [243] [DEBUG] [app] Starting request: urn:request:be34a03e-5969-4d1c-bbd2-06cc9dea8791 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:22,065 [242] [DEBUG] [app] Starting request: urn:request:4c9b0fea-fd38-4440-b440-e6be31735f7d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:22,065 [243] [DEBUG] [app] Ending request: urn:request:be34a03e-5969-4d1c-bbd2-06cc9dea8791 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be34a03e-5969-4d1c-bbd2-06cc9dea8791', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:22,065 [242] [DEBUG] [app] Ending request: urn:request:4c9b0fea-fd38-4440-b440-e6be31735f7d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c9b0fea-fd38-4440-b440-e6be31735f7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:22,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:23:22,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:22,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:22,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:22,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:22,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:22,070 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:22,070 [235] [DEBUG] [app] Starting request: urn:request:8259f740-4297-4c16-988d-128810b7fcea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:22,071 [236] [DEBUG] [app] Starting request: urn:request:091e3664-c8f1-4d35-bdf1-7a5332671c86 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:22,071 [235] [DEBUG] [app] Ending request: urn:request:8259f740-4297-4c16-988d-128810b7fcea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8259f740-4297-4c16-988d-128810b7fcea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:22,071 [236] [DEBUG] [app] Ending request: urn:request:091e3664-c8f1-4d35-bdf1-7a5332671c86 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:091e3664-c8f1-4d35-bdf1-7a5332671c86', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:22,071 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:22,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:22,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:22,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:22,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:22,072 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:22,079 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:22,079 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:22,088 [231] [DEBUG] [app] Ending request: urn:request:f566164a-8bed-46d1-822b-81f5bebdc02b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f566164a-8bed-46d1-822b-81f5bebdc02b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:22,088 [236] [DEBUG] [app] Ending request: urn:request:ead9331a-d61c-47e7-a884-55248b559ecc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ead9331a-d61c-47e7-a884-55248b559ecc', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:22,088 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:22,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:22,088 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:23:22,088 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.030) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) notificationworker stdout | 2023-01-17 06:23:25,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:25,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:35.646354+00:00 (in 9.999553 seconds) notificationworker stdout | 2023-01-17 06:23:25,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:35 UTC)" (scheduled at 2023-01-17 06:23:25.646354+00:00) notificationworker stdout | 2023-01-17 06:23:25,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:25,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 25, 647249), True, datetime.datetime(2023, 1, 17, 6, 23, 25, 647249), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:25,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:25,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:25,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:35 UTC)" executed successfully exportactionlogsworker stdout | 2023-01-17 06:23:28,964 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:23:28,964 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:35.853396+00:00 (in 6.889277 seconds) exportactionlogsworker stdout | 2023-01-17 06:23:28,964 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:28 UTC)" (scheduled at 2023-01-17 06:23:28.963662+00:00) exportactionlogsworker stdout | 2023-01-17 06:23:28,964 [65] [DEBUG] [workers.queueworker] Running watchdog. exportactionlogsworker stdout | 2023-01-17 06:23:28,964 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:28 UTC)" executed successfully builder stdout | 2023-01-17 06:23:31,021 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:31,033 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:31,033 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:31,044 [60] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:31,411 [233] [DEBUG] [app] Starting request: urn:request:5df13c4c-f9ad-4aeb-93e7-814c7f43c2af (/api/v1/repository/build_test/bitbucket/build/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:31,411 [233] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:31,411 [233] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:31,412 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:31,422 [233] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:23:31,422 [233] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:23:31,422 [233] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:31,422 [233] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:23:31,423 [233] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:31,423 [233] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:31,423 [233] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:31,423 [233] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:31,425 [233] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:23:31,432 [233] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:23:31,432 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:31,438 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s', ['bitbucket', 'build_test', 3]) gunicorn-web stdout | 2023-01-17 06:23:31,443 [233] [DEBUG] [app] Ending request: urn:request:5df13c4c-f9ad-4aeb-93e7-814c7f43c2af (/api/v1/repository/build_test/bitbucket/build/) {'endpoint': 'api.repositorybuildlist', 'request_id': 'urn:request:5df13c4c-f9ad-4aeb-93e7-814c7f43c2af', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/build/?limit=3', 'path': '/api/v1/repository/build_test/bitbucket/build/', 'parameters': {'limit': '3'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:23:31,444 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:31,444 [233] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:23:31 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:31 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.035 1694 0.035) gcworker stdout | 2023-01-17 06:23:31,547 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:23:31,625 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} gunicorn-web stdout | 2023-01-17 06:23:31,819 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:23:31,828 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:23:31,831 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:23:31,834 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:23:31,836 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} gunicorn-web stdout | 2023-01-17 06:23:32,057 [236] [DEBUG] [app] Starting request: urn:request:d1478306-bca5-4de0-80d9-7547464fda7e (/api/v1/repository/build_test/bitbucket/trigger/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:32,057 [236] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:32,057 [236] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:32,058 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:32,068 [236] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:23:32,068 [236] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:23:32,068 [236] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:32,068 [236] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:23:32,068 [236] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:32,069 [236] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:32,069 [236] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:32,069 [236] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:32,071 [236] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:23:32,077 [236] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:23:32,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:32,083 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") WHERE (("t4"."username" = %s) AND ("t3"."name" = %s))', ['build_test', 'bitbucket']) gunicorn-web stdout | 2023-01-17 06:23:32,087 [236] [DEBUG] [app] Ending request: urn:request:d1478306-bca5-4de0-80d9-7547464fda7e (/api/v1/repository/build_test/bitbucket/trigger/) {'endpoint': 'api.buildtriggerlist', 'request_id': 'urn:request:d1478306-bca5-4de0-80d9-7547464fda7e', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/', 'path': '/api/v1/repository/build_test/bitbucket/trigger/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:23:32,088 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:32,088 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:23:32 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/1.0" 200 1083 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:32 +0000] "GET /api/v1/repository/build_test/bitbucket/trigger/ HTTP/2.0" 200 1083 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.033 417 0.033) queuecleanupworker stdout | 2023-01-17 06:23:32,601 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:23:32,975 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:23:33,020 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:23:33,039 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:23:33,501 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} gunicorn-web stdout | 2023-01-17 06:23:33,999 [231] [DEBUG] [app] Starting request: urn:request:a84472bd-be51-4fa7-9e2f-a49b1fffba3d (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:33,999 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:34,000 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:34,001 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:34,011 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:23:34,011 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:23:34,012 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:34,012 [231] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:23:34,012 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:34,012 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:34,012 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:34,012 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:34,014 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:23:34,021 [231] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:23:34,022 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:34,027 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:34,031 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:34,036 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:34,036 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:23:34,036 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '138594161647168087391673936614'), ('oauth_timestamp', '1673936614'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=138594161647168087391673936614&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936614&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches-tags gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fbranches-tags&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D138594161647168087391673936614%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936614%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: OpF390bmUa2FAivad5bDSmR1RfQ= gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches-tags gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="138594161647168087391673936614", oauth_timestamp="1673936614", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="OpF390bmUa2FAivad5bDSmR1RfQ%3D"'} gunicorn-web stdout | 2023-01-17 06:23:34,037 [231] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:23:34,039 [231] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:23:34,123 [231] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/branches-tags HTTP/1.1" 200 178 gunicorn-web stdout | 2023-01-17 06:23:34,125 [231] [DEBUG] [app] Ending request: urn:request:a84472bd-be51-4fa7-9e2f-a49b1fffba3d (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs) {'endpoint': 'api.buildtriggerfieldvalues', 'request_id': 'urn:request:a84472bd-be51-4fa7-9e2f-a49b1fffba3d', 'remote_addr': '10.128.2.7', 'http_method': 'POST', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs', 'path': '/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs', 'parameters': {}, 'json_body': {}, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:23:34,126 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:34,126 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:23:34 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs HTTP/1.0" 200 51 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:34 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/fields/refs HTTP/2.0" 200 51 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.172 553 0.128) notificationworker stdout | 2023-01-17 06:23:35,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:35,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:36.635997+00:00 (in 0.989193 seconds) notificationworker stdout | 2023-01-17 06:23:35,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:45 UTC)" (scheduled at 2023-01-17 06:23:35.646354+00:00) notificationworker stdout | 2023-01-17 06:23:35,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:35,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 35, 647172), True, datetime.datetime(2023, 1, 17, 6, 23, 35, 647172), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:35,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:35,658 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:35,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:45 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:23:35,787 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:23:35,789 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:23:35,791 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} exportactionlogsworker stdout | 2023-01-17 06:23:35,853 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run exportactionlogsworker stdout | 2023-01-17 06:23:35,854 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:28.963662+00:00 (in 53.109796 seconds) exportactionlogsworker stdout | 2023-01-17 06:23:35,854 [65] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:35 UTC)" (scheduled at 2023-01-17 06:23:35.853396+00:00) exportactionlogsworker stdout | 2023-01-17 06:23:35,854 [65] [DEBUG] [workers.queueworker] Getting work item from queue. exportactionlogsworker stdout | 2023-01-17 06:23:35,854 [65] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 35, 854233), True, datetime.datetime(2023, 1, 17, 6, 23, 35, 854233), 0, 'exportactionlogs/%', 50, 1, 0]) exportactionlogsworker stdout | 2023-01-17 06:23:35,865 [65] [DEBUG] [workers.queueworker] No more work. exportactionlogsworker stdout | 2023-01-17 06:23:35,865 [65] [DEBUG] [data.database] Disconnecting from database. exportactionlogsworker stdout | 2023-01-17 06:23:35,865 [65] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:35 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:35,896 [231] [DEBUG] [app] Starting request: urn:request:82fd5f09-a46b-47d7-a56f-7dd1770e3c99 (/api/v1/repository/build_test/bitbucket/build/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:35,897 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:35,897 [231] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:35,898 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:35,907 [231] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:23:35,907 [231] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:23:35,907 [231] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:35,907 [231] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:23:35,908 [231] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:35,908 [231] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:35,908 [231] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:35,908 [231] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:35,910 [231] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:23:35,916 [231] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:23:35,918 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:35,923 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."access_token_id", "t1"."resource_key", "t1"."job_config", "t1"."phase", "t1"."started", "t1"."display_name", "t1"."trigger_id", "t1"."pull_robot_id", "t1"."logs_archived", "t1"."queue_id", "t2"."id", "t2"."uuid", "t2"."service_id", "t2"."repository_id", "t2"."connected_user_id", "t2"."secure_auth_token", "t2"."secure_private_key", "t2"."fully_migrated", "t2"."config", "t2"."write_token_id", "t2"."pull_robot_id", "t2"."enabled", "t2"."disabled_reason_id", "t2"."disabled_datetime", "t2"."successive_failure_count", "t2"."successive_internal_error_count", "t3"."id", "t3"."name", "t4"."id", "t4"."namespace_user_id", "t4"."name", "t4"."visibility_id", "t4"."description", "t4"."badge_token", "t4"."kind_id", "t4"."trust_enabled", "t4"."state", "t5"."id", "t5"."uuid", "t5"."username", "t5"."password_hash", "t5"."email", "t5"."verified", "t5"."stripe_id", "t5"."organization", "t5"."robot", "t5"."invoice_email", "t5"."invalid_login_attempts", "t5"."last_invalid_login", "t5"."removed_tag_expiration_s", "t5"."enabled", "t5"."invoice_email_address", "t5"."given_name", "t5"."family_name", "t5"."company", "t5"."location", "t5"."maximum_queued_builds_count", "t5"."creation_date", "t5"."last_accessed", "t6"."id", "t6"."uuid", "t6"."username", "t6"."password_hash", "t6"."email", "t6"."verified", "t6"."stripe_id", "t6"."organization", "t6"."robot", "t6"."invoice_email", "t6"."invalid_login_attempts", "t6"."last_invalid_login", "t6"."removed_tag_expiration_s", "t6"."enabled", "t6"."invoice_email_address", "t6"."given_name", "t6"."family_name", "t6"."company", "t6"."location", "t6"."maximum_queued_builds_count", "t6"."creation_date", "t6"."last_accessed" FROM "repositorybuild" AS "t1" INNER JOIN "repository" AS "t4" ON ("t1"."repository_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t4"."namespace_user_id" = "t5"."id") LEFT OUTER JOIN "user" AS "t6" ON ("t1"."pull_robot_id" = "t6"."id") LEFT OUTER JOIN "repositorybuildtrigger" AS "t2" ON ("t1"."trigger_id" = "t2"."id") LEFT OUTER JOIN "buildtriggerservice" AS "t3" ON ("t2"."service_id" = "t3"."id") WHERE (("t4"."name" = %s) AND ("t5"."username" = %s)) ORDER BY "t1"."started" DESC LIMIT %s', ['bitbucket', 'build_test', 3]) gunicorn-web stdout | 2023-01-17 06:23:35,928 [231] [DEBUG] [app] Ending request: urn:request:82fd5f09-a46b-47d7-a56f-7dd1770e3c99 (/api/v1/repository/build_test/bitbucket/build/) {'endpoint': 'api.repositorybuildlist', 'request_id': 'urn:request:82fd5f09-a46b-47d7-a56f-7dd1770e3c99', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/api/v1/repository/build_test/bitbucket/build/?limit=3', 'path': '/api/v1/repository/build_test/bitbucket/build/', 'parameters': {'limit': '3'}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:23:35,929 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:35,929 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:23:35 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/1.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:35 +0000] "GET /api/v1/repository/build_test/bitbucket/build/?limit=3 HTTP/2.0" 200 15 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.034 1694 0.034) gunicorn-registry stdout | 2023-01-17 06:23:36,297 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:23:36,299 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:23:36,303 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:23:36,313 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:23:36,316 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:23:36,318 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:23:36,324 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:23:36,329 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:23:36,331 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} gunicorn-web stdout | 2023-01-17 06:23:36,449 [235] [DEBUG] [app] Starting request: urn:request:134c7881-c5a5-42ed-9f40-c5264c2c3822 (/api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/start) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:36,450 [235] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:36,450 [235] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:36,450 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:36,466 [235] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:23:36,466 [235] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:23:36,466 [235] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:36,467 [235] [DEBUG] [endpoints.api] Checking permission for repo: build_test/bitbucket gunicorn-web stdout | 2023-01-17 06:23:36,467 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:36,467 [235] [DEBUG] [auth.permissions] User permission: _UserTypeNeed(type='user', username='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:36,467 [235] [DEBUG] [auth.permissions] User namespace permission: _NamespaceWideNeed(type='organization', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:36,467 [235] [DEBUG] [auth.permissions] User namespace repo permission: _NamespaceWideNeed(type='organizationrepo', namespace='user1', role='admin') gunicorn-web stdout | 2023-01-17 06:23:36,469 [235] [DEBUG] [peewee] ('(SELECT "t1"."id", "t1"."team_id", "t1"."user_id", "t1"."repository_id", "t1"."role_id", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorypermission" AS "t1" INNER JOIN "role" AS "t2" ON ("t1"."role_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."user_id" = "t5"."id") WHERE ((("t3"."name" = %s) AND ("t4"."username" = %s)) AND ("t5"."id" = %s))) UNION (SELECT "t6"."id", "t6"."team_id", "t6"."user_id", "t6"."repository_id", "t6"."role_id", "t7"."id", "t7"."name", "t8"."id", "t8"."namespace_user_id", "t8"."name", "t8"."visibility_id", "t8"."description", "t8"."badge_token", "t8"."kind_id", "t8"."trust_enabled", "t8"."state", "t9"."id", "t9"."uuid", "t9"."username", "t9"."password_hash", "t9"."email", "t9"."verified", "t9"."stripe_id", "t9"."organization", "t9"."robot", "t9"."invoice_email", "t9"."invalid_login_attempts", "t9"."last_invalid_login", "t9"."removed_tag_expiration_s", "t9"."enabled", "t9"."invoice_email_address", "t9"."given_name", "t9"."family_name", "t9"."company", "t9"."location", "t9"."maximum_queued_builds_count", "t9"."creation_date", "t9"."last_accessed" FROM "repositorypermission" AS "t6" INNER JOIN "role" AS "t7" ON ("t6"."role_id" = "t7"."id") INNER JOIN "repository" AS "t8" ON ("t6"."repository_id" = "t8"."id") INNER JOIN "user" AS "t9" ON ("t8"."namespace_user_id" = "t9"."id") INNER JOIN "team" AS "t10" ON ("t6"."team_id" = "t10"."id") INNER JOIN "teammember" AS "t11" ON ("t11"."team_id" = "t10"."id") INNER JOIN "user" AS "t12" ON ("t12"."id" = "t11"."user_id") WHERE ((("t8"."name" = %s) AND ("t9"."username" = %s)) AND ("t12"."id" = %s)))', ['bitbucket', 'build_test', 2, 'bitbucket', 'build_test', 2]) gunicorn-web stdout | 2023-01-17 06:23:36,476 [235] [DEBUG] [auth.permissions] User added permission: _ResourceNeed(type='repository', namespace='build_test', name='bitbucket', role='admin') gunicorn-web stdout | 2023-01-17 06:23:36,477 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") INNER JOIN "repositorykind" AS "t3" ON ("t1"."kind_id" = "t3"."id") WHERE (((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) AND ("t3"."name" = %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 'application', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:36,482 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:36,487 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."service_id", "t1"."repository_id", "t1"."connected_user_id", "t1"."secure_auth_token", "t1"."secure_private_key", "t1"."fully_migrated", "t1"."config", "t1"."write_token_id", "t1"."pull_robot_id", "t1"."enabled", "t1"."disabled_reason_id", "t1"."disabled_datetime", "t1"."successive_failure_count", "t1"."successive_internal_error_count", "t2"."id", "t2"."name", "t3"."id", "t3"."namespace_user_id", "t3"."name", "t3"."visibility_id", "t3"."description", "t3"."badge_token", "t3"."kind_id", "t3"."trust_enabled", "t3"."state", "t4"."id", "t4"."uuid", "t4"."username", "t4"."password_hash", "t4"."email", "t4"."verified", "t4"."stripe_id", "t4"."organization", "t4"."robot", "t4"."invoice_email", "t4"."invalid_login_attempts", "t4"."last_invalid_login", "t4"."removed_tag_expiration_s", "t4"."enabled", "t4"."invoice_email_address", "t4"."given_name", "t4"."family_name", "t4"."company", "t4"."location", "t4"."maximum_queued_builds_count", "t4"."creation_date", "t4"."last_accessed" FROM "repositorybuildtrigger" AS "t1" INNER JOIN "buildtriggerservice" AS "t2" ON ("t1"."service_id" = "t2"."id") INNER JOIN "repository" AS "t3" ON ("t1"."repository_id" = "t3"."id") INNER JOIN "user" AS "t4" ON ("t3"."namespace_user_id" = "t4"."id") INNER JOIN "user" AS "t5" ON ("t1"."connected_user_id" = "t5"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['03697f15-5e57-4d5a-ae56-cb0fcc4b4f44', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:36,492 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'bitbucket', 3, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '17423006574903512311673936616'), ('oauth_timestamp', '1673936616'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=17423006574903512311673936616&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936616&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/main-branch gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fmain-branch&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D17423006574903512311673936616%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936616%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: UMsnCpgYuGFQwiQcPi8usxLdD4U= gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/main-branch gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="17423006574903512311673936616", oauth_timestamp="1673936616", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="UMsnCpgYuGFQwiQcPi8usxLdD4U%3D"'} gunicorn-web stdout | 2023-01-17 06:23:36,496 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:23:36,498 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:23:36,613 [235] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/main-branch HTTP/1.1" 200 18 gunicorn-web stdout | 2023-01-17 06:23:36,615 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:23:36,615 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:23:36,615 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '134158850868860987121673936616'), ('oauth_timestamp', '1673936616'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=134158850868860987121673936616&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936616&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://api.bitbucket.org/2.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/refs/branches/master gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fapi.bitbucket.org%2F2.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Frefs%2Fbranches%2Fmaster&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D134158850868860987121673936616%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936616%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: Sd5XmmBSWPwHi5Mc2h+0AwSqo3c= gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://api.bitbucket.org/2.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/refs/branches/master gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="134158850868860987121673936616", oauth_timestamp="1673936616", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="Sd5XmmBSWPwHi5Mc2h%2B0AwSqo3c%3D"'} gunicorn-web stdout | 2023-01-17 06:23:36,616 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:23:36,617 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): api.bitbucket.org:443 notificationworker stdout | 2023-01-17 06:23:36,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:36,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:45.646354+00:00 (in 9.009938 seconds) notificationworker stdout | 2023-01-17 06:23:36,636 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:28:36 UTC)" (scheduled at 2023-01-17 06:23:36.635997+00:00) notificationworker stdout | 2023-01-17 06:23:36,637 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2023, 1, 17, 6, 23, 36, 636781), 'notification/%']) notificationworker stdout | 2023-01-17 06:23:36,647 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 23, 36, 636781), True, datetime.datetime(2023, 1, 17, 6, 23, 36, 636781), 0, 'notification/%']) notificationworker stdout | 2023-01-17 06:23:36,651 [76] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2023, 1, 17, 6, 23, 36, 636781), True, datetime.datetime(2023, 1, 17, 6, 23, 36, 636781), 0, 'notification/%', False, datetime.datetime(2023, 1, 17, 6, 23, 36, 636781), 'notification/%']) notificationworker stdout | 2023-01-17 06:23:36,655 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:36,655 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2023-01-17 06:28:36 UTC)" executed successfully buildlogsarchiver stdout | 2023-01-17 06:23:36,675 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} gunicorn-web stdout | 2023-01-17 06:23:36,773 [235] [DEBUG] [urllib3.connectionpool] https://api.bitbucket.org:443 "GET /2.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/refs/branches/master HTTP/1.1" 200 3036 gunicorn-web stdout | 2023-01-17 06:23:36,774 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '86793981095719835331673936616'), ('oauth_timestamp', '1673936616'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=86793981095719835331673936616&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936616&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/changesets/fee267ee19397002d6ac895968339f0b68bd5c7f gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Frepositories%2F%257B35aa643d-9d7c-4c91-970f-1915ef761a21%257D%2Fquay_build_test%2Fchangesets%2Ffee267ee19397002d6ac895968339f0b68bd5c7f&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D86793981095719835331673936616%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936616%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: /+uWeA94nVAjUQo1LkwvZOxoIBg= gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/changesets/fee267ee19397002d6ac895968339f0b68bd5c7f gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="86793981095719835331673936616", oauth_timestamp="1673936616", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="%2F%2BuWeA94nVAjUQo1LkwvZOxoIBg%3D"'} gunicorn-web stdout | 2023-01-17 06:23:36,775 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:23:36,776 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 expiredappspecifictokenworker stdout | 2023-01-17 06:23:36,783 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:23:36,905 [235] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/repositories/%7B35aa643d-9d7c-4c91-970f-1915ef761a21%7D/quay_build_test/changesets/fee267ee19397002d6ac895968339f0b68bd5c7f HTTP/1.1" 200 417 gunicorn-web stdout | 2023-01-17 06:23:36,931 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Signing request using client gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Including body in call to sign: False gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Collected params: [('oauth_nonce', '94690642231915079951673936616'), ('oauth_timestamp', '1673936616'), ('oauth_version', '1.0'), ('oauth_signature_method', 'HMAC-SHA1'), ('oauth_consumer_key', 'UYxg8VHU7BKPrrsbhd'), ('oauth_token', 'QyVRmX48G972D8vQsM')] gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized params: oauth_consumer_key=UYxg8VHU7BKPrrsbhd&oauth_nonce=94690642231915079951673936616&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1673936616&oauth_token=QyVRmX48G972D8vQsM&oauth_version=1.0 gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Normalized URI: https://bitbucket.org/!api/1.0/users/hu_weihua@163.com gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signing: signature base string: GET&https%3A%2F%2Fbitbucket.org%2F%21api%2F1.0%2Fusers%2Fhu_weihua%40163.com&oauth_consumer_key%3DUYxg8VHU7BKPrrsbhd%26oauth_nonce%3D94690642231915079951673936616%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1673936616%26oauth_token%3DQyVRmX48G972D8vQsM%26oauth_version%3D1.0 gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Signature: +nSiNxbKi6Yhn9CVEiMj5x31znk= gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [oauthlib.oauth1.rfc5849] Encoding URI, headers and body to utf-8. gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated url: https://bitbucket.org/!api/1.0/users/hu_weihua@163.com gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated headers: {b'Authorization': b'OAuth oauth_nonce="94690642231915079951673936616", oauth_timestamp="1673936616", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="UYxg8VHU7BKPrrsbhd", oauth_token="QyVRmX48G972D8vQsM", oauth_signature="%2BnSiNxbKi6Yhn9CVEiMj5x31znk%3D"'} gunicorn-web stdout | 2023-01-17 06:23:36,932 [235] [DEBUG] [requests_oauthlib.oauth1_auth] Updated body: None gunicorn-web stdout | 2023-01-17 06:23:36,933 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): bitbucket.org:443 gunicorn-web stdout | 2023-01-17 06:23:37,047 [235] [DEBUG] [urllib3.connectionpool] https://bitbucket.org:443 "GET /!api/1.0/users/hu_weihua@163.com HTTP/1.1" 200 1242 gunicorn-web stdout | 2023-01-17 06:23:37,059 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:37,059 [231] [DEBUG] [app] Starting request: urn:request:3dff4dbb-b76d-461a-ae33-8f6c19a2b565 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:37,060 [236] [DEBUG] [app] Starting request: urn:request:a0c29162-e67d-4c59-84f8-9df8301fb4c5 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:37,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,061 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,062 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "role" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['write', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:37,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:37,064 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:37,065 [230] [DEBUG] [app] Starting request: urn:request:8ca1b97a-e8b3-416b-9335-8238d1a75e76 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:37,065 [243] [DEBUG] [app] Starting request: urn:request:7b9ab8d6-52b7-4159-a637-8db81ea65362 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:37,065 [230] [DEBUG] [app] Ending request: urn:request:8ca1b97a-e8b3-416b-9335-8238d1a75e76 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8ca1b97a-e8b3-416b-9335-8238d1a75e76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:37,066 [243] [DEBUG] [app] Ending request: urn:request:7b9ab8d6-52b7-4159-a637-8db81ea65362 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b9ab8d6-52b7-4159-a637-8db81ea65362', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:37,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:23:37,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:23:37,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:37,066 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "accesstokenkind" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['build-worker', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:37,066 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:37,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,067 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:37,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:37,069 [236] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:37,070 [233] [DEBUG] [app] Starting request: urn:request:8e952af4-6bed-4e19-9610-bcf694c2ec7e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:37,070 [231] [DEBUG] [app] Starting request: urn:request:203bb33d-73c1-4d48-aaa2-a26c65480691 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:37,070 [231] [DEBUG] [app] Ending request: urn:request:203bb33d-73c1-4d48-aaa2-a26c65480691 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:203bb33d-73c1-4d48-aaa2-a26c65480691', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:37,070 [233] [DEBUG] [app] Ending request: urn:request:8e952af4-6bed-4e19-9610-bcf694c2ec7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e952af4-6bed-4e19-9610-bcf694c2ec7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:37,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:37,071 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:23:37,071 [236] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:37,071 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:37,071 [236] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:37,071 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:37,073 [235] [DEBUG] [peewee] ('INSERT INTO "accesstoken" ("friendly_name", "token_name", "token_code", "repository_id", "created", "role_id", "temporary", "kind_id") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "accesstoken"."id"', ['Repository Build Token', 'T9OAS2J03ZEC4QRUINF2UXYA3BRCMPE1', 'v0$$MwNoKPR9djhW23EYXJ0xZl+t88/FmMwufZXXCrrEI9Q7qtTKytmt8YgxjF63reIJLuuJnnJQGMMfnic7JA==', 1, datetime.datetime(2023, 1, 17, 6, 23, 37, 73007), 2, True, 1]) gunicorn-web stdout | 2023-01-17 06:23:37,078 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:37,078 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:37,078 [235] [DEBUG] [endpoints.building] Creating build fee267e with repo quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/build_test/bitbucket tags ['master', 'latest'] gunicorn-web stdout | 2023-01-17 06:23:37,079 [235] [DEBUG] [peewee] ('INSERT INTO "repositorybuild" ("uuid", "repository_id", "access_token_id", "resource_key", "job_config", "phase", "started", "display_name", "trigger_id", "pull_robot_id", "logs_archived") VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) RETURNING "repositorybuild"."id"', ['582cef57-43a5-43d1-9ca7-1d229e4a10d8', 1, 2, None, '{"docker_tags": ["master", "latest"], "registry": "quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com", "build_subdir": "//Dockerfile", "context": "/", "trigger_metadata": {"commit": "fee267ee1939", "ref": "refs/heads/master", "default_branch": "master", "git_url": "git@bitbucket.org:{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test.git", "commit_info": {"url": "https://bitbucket.org/{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test/commits/fee267ee1939", "message": "filename1 edited online with Bitbucket", "date": "2022-12-09 08:08:42", "author": {"username": "hu_weihua", "avatar_url": "https://secure.gravatar.com/avatar/244e4d69a3c88a18cb1b2839da515ec7?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FH-0.png"}}}, "is_manual": true, "manual_user": "user1", "archive_url": null}', 'waiting', datetime.datetime(2023, 1, 17, 6, 23, 37, 78844), 'fee267e', 1, None, False]) gunicorn-web stdout | 2023-01-17 06:23:37,082 [235] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['dockerfilebuild/build_test/bitbucket/', '{"build_uuid": "582cef57-43a5-43d1-9ca7-1d229e4a10d8", "pull_credentials": null}', datetime.datetime(2023, 1, 17, 6, 23, 37, 82425), True, 3, 'c98c4e88-d474-4d13-92f7-5fb33aa143db']) gunicorn-web stdout | 2023-01-17 06:23:37,084 [235] [DEBUG] [peewee] ('UPDATE "repositorybuild" SET "uuid" = %s, "repository_id" = %s, "access_token_id" = %s, "resource_key" = %s, "job_config" = %s, "phase" = %s, "started" = %s, "display_name" = %s, "trigger_id" = %s, "pull_robot_id" = %s, "logs_archived" = %s, "queue_id" = %s WHERE ("repositorybuild"."id" = %s)', ['582cef57-43a5-43d1-9ca7-1d229e4a10d8', 1, 2, None, '{"docker_tags": ["master", "latest"], "registry": "quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com", "build_subdir": "//Dockerfile", "context": "/", "trigger_metadata": {"commit": "fee267ee1939", "ref": "refs/heads/master", "default_branch": "master", "git_url": "git@bitbucket.org:{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test.git", "commit_info": {"url": "https://bitbucket.org/{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test/commits/fee267ee1939", "message": "filename1 edited online with Bitbucket", "date": "2022-12-09 08:08:42", "author": {"username": "hu_weihua", "avatar_url": "https://secure.gravatar.com/avatar/244e4d69a3c88a18cb1b2839da515ec7?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FH-0.png"}}}, "is_manual": true, "manual_user": "user1", "archive_url": null}', 'waiting', datetime.datetime(2023, 1, 17, 6, 23, 37, 78844), 'fee267e', 1, None, False, '1', 1]) gunicorn-web stdout | 2023-01-17 06:23:37,087 [236] [DEBUG] [app] Ending request: urn:request:a0c29162-e67d-4c59-84f8-9df8301fb4c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0c29162-e67d-4c59-84f8-9df8301fb4c5', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:37,087 [236] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:37,087 [231] [DEBUG] [app] Ending request: urn:request:3dff4dbb-b76d-461a-ae33-8f6c19a2b565 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3dff4dbb-b76d-461a-ae33-8f6c19a2b565', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:37,087 [231] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:37,087 [236] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.028 47 0.029) gunicorn-web stdout | 2023-01-17 06:23:37,087 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) gunicorn-web stdout | 2023-01-17 06:23:37,093 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:37,096 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "logentrykind" AS "t1"', []) gunicorn-web stdout | 2023-01-17 06:23:37,100 [235] [DEBUG] [peewee] ('INSERT INTO "logentry3" ("kind_id", "account_id", "performer_id", "repository_id", "datetime", "ip", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "logentry3"."id"', [9, 3, None, 1, datetime.datetime(2023, 1, 17, 6, 23, 37, 92695), '10.128.2.7', '{"build_id": "582cef57-43a5-43d1-9ca7-1d229e4a10d8", "docker_tags": ["master", "latest"], "repo": "bitbucket", "namespace": "build_test", "is_manual": true, "manual_user": "user1", "trigger_id": "03697f15-5e57-4d5a-ae56-cb0fcc4b4f44", "trigger_kind": "bitbucket", "trigger_metadata": {"commit": "fee267ee1939", "ref": "refs/heads/master", "default_branch": "master", "git_url": "git@bitbucket.org:{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test.git", "commit_info": {"url": "https://bitbucket.org/{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test/commits/fee267ee1939", "message": "filename1 edited online with Bitbucket", "date": "2022-12-09 08:08:42", "author": {"username": "hu_weihua", "avatar_url": "https://secure.gravatar.com/avatar/244e4d69a3c88a18cb1b2839da515ec7?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FH-0.png"}}}}']) gunicorn-web stdout | 2023-01-17 06:23:37,107 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'bitbucket', 'build_queued']) gunicorn-web stdout | 2023-01-17 06:23:37,115 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:37,115 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:37,115 [235] [DEBUG] [auth.permissions] Loading user permissions after deferring for: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:37,116 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:37,119 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:37,120 [235] [ERROR] [gunicorn.error] Error handling request /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/start gunicorn-web stdout | Traceback (most recent call last): gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-web stdout | self.handle_request(listener_name, req, client, addr) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 127, in handle_request gunicorn-web stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request gunicorn-web stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2463, in __call__ gunicorn-web stdout | return self.wsgi_app(environ, start_response) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__ gunicorn-web stdout | return self.app(environ, start_response) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2449, in wsgi_app gunicorn-web stdout | response = self.handle_exception(e) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router gunicorn-web stdout | return original_handler(e) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1866, in handle_exception gunicorn-web stdout | reraise(exc_type, exc_value, tb) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 38, in reraise gunicorn-web stdout | raise value.with_traceback(tb) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2446, in wsgi_app gunicorn-web stdout | response = self.full_dispatch_request() gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1951, in full_dispatch_request gunicorn-web stdout | rv = self.handle_user_exception(e) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 271, in error_router gunicorn-web stdout | return original_handler(e) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1820, in handle_user_exception gunicorn-web stdout | reraise(exc_type, exc_value, tb) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 38, in reraise gunicorn-web stdout | raise value.with_traceback(tb) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1949, in full_dispatch_request gunicorn-web stdout | rv = self.dispatch_request() gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1935, in dispatch_request gunicorn-web stdout | return self.view_functions[rule.endpoint](**req.view_args) gunicorn-web stdout | File "/quay-registry/endpoints/decorators.py", line 256, in wrapper gunicorn-web stdout | return func(*args, **kwargs) gunicorn-web stdout | File "/quay-registry/auth/decorators.py", line 65, in wrapper gunicorn-web stdout | return func(*args, **kwargs) gunicorn-web stdout | File "/quay-registry/util/request.py", line 50, in wrapper gunicorn-web stdout | return decorator(func)(*args, **kwargs) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask_restful/utils/cors.py", line 35, in wrapped_function gunicorn-web stdout | resp = make_response(f(*args, **kwargs)) gunicorn-web stdout | File "/quay-registry/endpoints/csrf.py", line 73, in wrapper gunicorn-web stdout | resp = func(*args, **kwargs) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 467, in wrapper gunicorn-web stdout | resp = resource(*args, **kwargs) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask/views.py", line 89, in view gunicorn-web stdout | return self.dispatch_request(*args, **kwargs) gunicorn-web stdout | File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 582, in dispatch_request gunicorn-web stdout | resp = meth(*args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/decorators.py", line 189, in wrapper gunicorn-web stdout | return func(*args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/api/__init__.py", line 247, in wrapper gunicorn-web stdout | return func(namespace, repository, *args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/decorators.py", line 164, in wrapper gunicorn-web stdout | return func(*args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/api/__init__.py", line 342, in wrapped gunicorn-web stdout | return func(self, namespace, repository, *args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/api/__init__.py", line 296, in wrapped gunicorn-web stdout | return func(self, namespace_name, repository_name, *args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/api/__init__.py", line 308, in wrapped gunicorn-web stdout | return func(self, namespace_name, repository_name, *args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/api/__init__.py", line 283, in wrapped gunicorn-web stdout | return func(self, namespace_name, repository_name, *args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/api/__init__.py", line 494, in wrapped gunicorn-web stdout | return func(self, *args, **kwargs) gunicorn-web stdout | File "/quay-registry/endpoints/api/trigger.py", line 488, in post gunicorn-web stdout | log_action("start_build_trigger", namespace_name, {"trigger_uuid": trigger_uuid}) gunicorn-web stdout | File "/quay-registry/endpoints/api/__init__.py", line 528, in log_action gunicorn-web stdout | logs_model.log_action( gunicorn-web stdout | File "/quay-registry/data/logs_model/table_logs_model.py", line 229, in log_action gunicorn-web stdout | model.log.log_action( gunicorn-web stdout | File "/quay-registry/data/model/log.py", line 255, in log_action gunicorn-web stdout | kind = _get_log_entry_kind(kind_name) gunicorn-web stdout | File "/quay-registry/data/model/log.py", line 123, in _get_log_entry_kind gunicorn-web stdout | return kinds[name] gunicorn-web stdout | KeyError: 'start_build_trigger' gunicorn-web stdout | 2023-01-17 06:23:37,121 [235] [INFO] [gunicorn.access] - - [17/Jan/2023:06:23:37 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/start HTTP/1.0" 500 0 "-" "-" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:37 +0000] "POST /api/v1/repository/build_test/bitbucket/trigger/03697f15-5e57-4d5a-ae56-cb0fcc4b4f44/start HTTP/2.0" 500 141 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.673 494 0.673) securityscanningnotificationworker stdout | 2023-01-17 06:23:37,256 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:23:37,257 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:37.537407+00:00 (in 0.280376 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:23:37,257 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:37 UTC)" (scheduled at 2023-01-17 06:23:37.256652+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:23:37,257 [81] [DEBUG] [workers.queueworker] Getting work item from queue. securityscanningnotificationworker stdout | 2023-01-17 06:23:37,258 [81] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 37, 257406), True, datetime.datetime(2023, 1, 17, 6, 23, 37, 257406), 0, 'secscanv4/%', 50, 1, 0]) securityscanningnotificationworker stdout | 2023-01-17 06:23:37,268 [81] [DEBUG] [workers.queueworker] No more work. securityscanningnotificationworker stdout | 2023-01-17 06:23:37,268 [81] [DEBUG] [data.database] Disconnecting from database. securityscanningnotificationworker stdout | 2023-01-17 06:23:37,268 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:37 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:37,509 [231] [DEBUG] [app] Starting request: urn:request:5b79e9a2-8898-4f71-8bb8-2e6233497068 (/500) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:37,510 [231] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:37,514 [231] [DEBUG] [app] Ending request: urn:request:5b79e9a2-8898-4f71-8bb8-2e6233497068 (/500) {'endpoint': 'web.internal_error_display', 'request_id': 'urn:request:5b79e9a2-8898-4f71-8bb8-2e6233497068', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500', 'path': '/500', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:23:37,515 [231] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:23:37 +0000] "GET /500 HTTP/1.0" 200 2629 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /500 HTTP/2.0" 200 980 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/repository/build_test/bitbucket?tab=builds" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.007 191 0.006) securityscanningnotificationworker stdout | 2023-01-17 06:23:37,537 [81] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityscanningnotificationworker stdout | 2023-01-17 06:23:37,537 [81] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:37.256652+00:00 (in 59.718860 seconds) securityscanningnotificationworker stdout | 2023-01-17 06:23:37,538 [81] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:37 UTC)" (scheduled at 2023-01-17 06:23:37.537407+00:00) securityscanningnotificationworker stdout | 2023-01-17 06:23:37,538 [81] [DEBUG] [workers.queueworker] Running watchdog. securityscanningnotificationworker stdout | 2023-01-17 06:23:37,538 [81] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:37 UTC)" executed successfully nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /static/img/500/background.svg HTTP/2.0" 200 1126 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 660 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /static/img/500/ship.svg HTTP/2.0" 200 1497 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 956 -) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /static/img/500/water.svg HTTP/2.0" 200 1424 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.000 51 -) storagereplication stdout | 2023-01-17 06:23:37,902 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} gunicorn-web stdout | 2023-01-17 06:23:37,915 [236] [DEBUG] [app] Starting request: urn:request:2ff4aeaa-0c3c-4145-92e2-0ab9db1111e2 (/favicon.ico) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:37,916 [236] [DEBUG] [app] Ending request: urn:request:2ff4aeaa-0c3c-4145-92e2-0ab9db1111e2 (/favicon.ico) {'endpoint': None, 'request_id': 'urn:request:2ff4aeaa-0c3c-4145-92e2-0ab9db1111e2', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/favicon.ico', 'path': '/favicon.ico', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:37 +0000] "GET /favicon.ico HTTP/2.0" 308 415 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.003 42 0.003) gunicorn-web stdout | 2023-01-17 06:23:37,917 [236] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:23:37 +0000] "GET /favicon.ico HTTP/1.0" 308 415 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" servicekey stdout | 2023-01-17 06:23:37,940 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} gunicorn-web stdout | 2023-01-17 06:23:38,172 [233] [DEBUG] [app] Starting request: urn:request:bcebd098-fda4-4804-861a-813a7df1a8fd (/favicon.ico/) {'X-Forwarded-For': '10.128.2.7'} gunicorn-web stdout | 2023-01-17 06:23:38,172 [233] [DEBUG] [app] User loader loading deferred user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:38,172 [233] [DEBUG] [auth.cookie] Loading user from cookie: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:38,173 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['e07aafb9-6cbb-4287-9ea6-65bf7c105028', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:38,183 [233] [DEBUG] [auth.decorators] Found valid auth result: (, False, None, [, None, None, None, None, None]) gunicorn-web stdout | 2023-01-17 06:23:38,183 [233] [DEBUG] [auth.permissions] Identity loaded: gunicorn-web stdout | 2023-01-17 06:23:38,183 [233] [DEBUG] [auth.permissions] Deferring permissions for user with uuid: e07aafb9-6cbb-4287-9ea6-65bf7c105028 gunicorn-web stdout | 2023-01-17 06:23:38,184 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['favicon.ico', False, 1, 0]) gunicorn-web stdout | 2023-01-17 06:23:38,195 [233] [DEBUG] [endpoints.api.discovery] Unable to find method for POST in class gunicorn-web stdout | 2023-01-17 06:23:38,216 [233] [DEBUG] [app] Ending request: urn:request:bcebd098-fda4-4804-861a-813a7df1a8fd (/favicon.ico/) {'endpoint': 'web.redirect_to_namespace', 'request_id': 'urn:request:bcebd098-fda4-4804-861a-813a7df1a8fd', 'remote_addr': '10.128.2.7', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/favicon.ico/', 'path': '/favicon.ico/', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} gunicorn-web stdout | 2023-01-17 06:23:38,217 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:38,217 [233] [INFO] [gunicorn.access] 10.128.2.7 - - [17/Jan/2023:06:23:38 +0000] "GET /favicon.ico/ HTTP/1.0" 404 85004 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:38 +0000] "GET /favicon.ico/ HTTP/2.0" 404 13278 "https://quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com/500" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" (0.046 42 0.047) repositorygcworker stdout | 2023-01-17 06:23:38,406 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:23:38,789 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:23:38,822 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:23:39,073 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:23:39,279 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:23:39,722 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:23:40,324 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} exportactionlogsworker stdout | 2023-01-17 06:23:40,428 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:23:40,668 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:23:40,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:23:40,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:10.738568+00:00 (in 29.999532 seconds) buildlogsarchiver stdout | 2023-01-17 06:23:40,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:10 UTC)" (scheduled at 2023-01-17 06:23:40.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:23:40,740 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 23, 40, 739428), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:23:40,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:23:40,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:23:40,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:10 UTC)" executed successfully builder stdout | 2023-01-17 06:23:41,054 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:41,067 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:41,067 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:41,077 [60] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:23:41,148 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:23:41,148 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:42.941536+00:00 (in 1.792802 seconds) namespacegcworker stdout | 2023-01-17 06:23:41,149 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:41 UTC)" (scheduled at 2023-01-17 06:23:41.148245+00:00) namespacegcworker stdout | 2023-01-17 06:23:41,149 [74] [DEBUG] [workers.queueworker] Getting work item from queue. namespacegcworker stdout | 2023-01-17 06:23:41,150 [74] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 41, 149138), True, datetime.datetime(2023, 1, 17, 6, 23, 41, 149138), 0, 'namespacegc/%', 50, 1, 0]) namespacegcworker stdout | 2023-01-17 06:23:41,160 [74] [DEBUG] [workers.queueworker] No more work. namespacegcworker stdout | 2023-01-17 06:23:41,160 [74] [DEBUG] [data.database] Disconnecting from database. namespacegcworker stdout | 2023-01-17 06:23:41,160 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:41 UTC)" executed successfully namespacegcworker stdout | 2023-01-17 06:23:42,941 [74] [DEBUG] [apscheduler.scheduler] Looking for jobs to run namespacegcworker stdout | 2023-01-17 06:23:42,942 [74] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:41.148245+00:00 (in 58.206233 seconds) namespacegcworker stdout | 2023-01-17 06:23:42,942 [74] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:42 UTC)" (scheduled at 2023-01-17 06:23:42.941536+00:00) namespacegcworker stdout | 2023-01-17 06:23:42,942 [74] [DEBUG] [workers.queueworker] Running watchdog. namespacegcworker stdout | 2023-01-17 06:23:42,942 [74] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:24:42 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:23:45,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:45,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:55.646354+00:00 (in 9.999554 seconds) notificationworker stdout | 2023-01-17 06:23:45,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:55 UTC)" (scheduled at 2023-01-17 06:23:45.646354+00:00) notificationworker stdout | 2023-01-17 06:23:45,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:45,647 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 45, 647212), True, datetime.datetime(2023, 1, 17, 6, 23, 45, 647212), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:45,658 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:45,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:45,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:23:55 UTC)" executed successfully securityworker stdout | 2023-01-17 06:23:46,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:46,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:23:49.537472+00:00 (in 2.600002 seconds) securityworker stdout | 2023-01-17 06:23:46,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:16 UTC)" (scheduled at 2023-01-17 06:23:46.936988+00:00) securityworker stdout | 2023-01-17 06:23:46,937 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:46,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:23:46,940 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:46,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:46,952 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:46,955 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:23:46,955 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:16 UTC)" executed successfully nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:48 +0000] "POST /buildman_pb.BuildManager/Ping HTTP/2.0" 200 11 "-" "grpc-go/1.41.0" (0.005 156 0.005) builder stdout | 2023-01-17 06:23:48,115 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:23:48,130 [60] [DEBUG] [buildman.manager.ephemeral] Started build job buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8 with arguments {'build_package': '', 'context': '/', 'dockerfile_path': 'Dockerfile', 'repository': 'build_test/bitbucket', 'registry': 'quayregistry-quay-quay-enterprise.apps.whu411az22.qe.azure.devcluster.openshift.com', 'pull_token': 'T9OAS2J03ZEC4QRUINF2UXYA3BRCMPE1NZP7SXIWCBM51AZZRLXQQHZ5I6FTMLCS', 'push_token': 'T9OAS2J03ZEC4QRUINF2UXYA3BRCMPE1NZP7SXIWCBM51AZZRLXQQHZ5I6FTMLCS', 'tag_names': ['master', 'latest'], 'base_image': {}, 'git': {'url': 'git@bitbucket.org:{35aa643d-9d7c-4c91-970f-1915ef761a21}/quay_build_test.git', 'sha': 'fee267ee1939', 'private_key': '-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEAxolDGkD1OpihoZcn/HTyZYidKnzds01hRateQcxLeOEapWQi\npiKsdn3uQCW2uEB0UM26E+UgwcN2ZObUbcrpKQvuSTWdSDD4Fdo9Rh9MD/S9IQ2t\nS9QSAfBhF7v2n8O00P0k8hwjqADCI6GbjlyQP87FLVrOrP3zk3ax0CZ3RVYoke5o\nsjmDrKQCdqV7FJcburAJFmUcjb/ECPN/PP7sK0qViOKrjZ6OvgE5rJ7gHDOcUAJs\nGJ2XhRimzKKhwLd0KtAM+IhlaZv8p/rrRynT6CHuhZX+Bw9aRLT0JSFux8zMZzgq\nc9NaOs9ewn74VKMbrAJqmyBkT0C+IBkdjSZ8xQIDAQABAoIBAFQg+CFY41GoOd2a\niuQsvkaOJGNLtDp1kMGQA4oIQvb/FGZ/TzttN/2iEWJ3WL0s+tz+wFoL3tycsj/B\nnMTFlhXBq4C0Smi773R7HTnoqzebssUd4ZBfVh4iOVJwYOqEaMuiaZR6nhVj9OBa\na+tgh/Ox79XvQ+MEXcHgKysUEbh8MbUqPFGmTiONGcKeq7jD+V4pDcvzVR2Iyqta\nKuXMv40XJQOUD8uIkC+91vm459LIXesfyhhWYkpuJVm9lCpXn1EaU1IbzCmYqz9R\njTeO01ucLb9sH/QsZX2yLn9pxvcp+KQpZWJuQ2tVqnDKMsdbR38qQ1vzUdqJCXHn\nTTOl2eECgYEA7V+RuqskbIxjdwjcdYDnlQDnNeiS79xjCQOtE5lmQjTl3C0v2ccI\nLHCnhxPVMGZBhzXqMI2/GE+3e1aukehnkCvz2aUhD58kmNdy82zWLvzP9j8lxhuC\n0VxqfklTWNzMa5+xHL//XRJOVxbsSycSIjQh1mt9V9rDlXrHneatZTkCgYEA1h2F\nJLkQBbISyqZyIvqlpv0EvBZU6pllSIA/fskgiaUQSwvEMUC8Myg/JtVx7HGEThgF\n82Y2hCRZdMPsBkI7UZDx7udpKGfjUZADsyXxknEb4hAWtwl6NkYuhv1nK2pchChm\n2727Un7JOrd3w17T/hUf1r/EZu75CKZYCuks/+0CgYBu+FS6fbPDg0YkqeHWlPKu\nuANfMmEu/UrXNf69wOMLZItpCT7rgp6RmADZpy3tOIC94ziehkWl6e6PHJDz6UAx\nKDsRrp7sHKwRP/efUWw3/zyBlDfmi4n2skueykzKqD5wufEkzaVmL16sXC0PXS6f\nCtwez5WDcuQoh4O2j9bUEQKBgB7LrCS2AVnwXMeM3Qmu5zh3V8D9rUT4vxqIZ5mC\nsA1cElXY4WT8HJV3Cm45cONDCHZ6+hAh+WFUIRGsRDw6NnHq2QE1g2huWMGYZw/T\nO0WGwy3mhn3Y/RzlRzffJoGWEKxpazHbvh+FYMfzL2SINRFsVGH8vvDXGjZmRm97\nTh+ZAoGAdJxEiij9HdQMTpMxtr4tt9lrr/zMYvoOQNYFoVAHuYs1tGozKuW2ztNY\nh9pFvxfvvtybTNFgqRmJtnOTdyA67zZhRmp0LBNnZzFnoST99rI3LCT1u7fdWyBX\nlBUmr20tSc0jLDXpmjXouXpXlMJZmiePLu9wyNwz7z+YnMxIb5c=\n-----END RSA PRIVATE KEY-----\n'}} nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:48 +0000] "POST /buildman_pb.BuildManager/RegisterBuildJob HTTP/2.0" 200 3022 "-" "grpc-go/1.41.0" (0.041 1032 0.040) builder stdout | 2023-01-17 06:23:48,147 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:23:48.134995\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:23:48,150 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:23:48,164 [60] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:48 +0000] "POST /buildman_pb.BuildManager/SetPhase HTTP/2.0" 200 9 "-" "grpc-go/1.41.0" (0.035 1025 0.034) builder stdout | 2023-01-17 06:23:48,564 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:23:48,584 [60] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:23:48 +0000] "POST /buildman_pb.BuildManager/SetPhase HTTP/2.0" 200 9 "-" "grpc-go/1.41.0" (0.039 1129 0.040) securityworker stdout | 2023-01-17 06:23:49,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:23:49,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:16.936988+00:00 (in 27.399021 seconds) securityworker stdout | 2023-01-17 06:23:49,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:19 UTC)" (scheduled at 2023-01-17 06:23:49.537472+00:00) securityworker stdout | 2023-01-17 06:23:49,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:49,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:23:49,541 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:49,541 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:23:49,541 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:49,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:23:49,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:23:49,544 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:23:49,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:23:49,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:49,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:23:49,555 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:23:49,555 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:23:49,555 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:23:49,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:23:49,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:23:49,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:23:49,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:19 UTC)" executed successfully builder stdout | 2023-01-17 06:23:50,164 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:23:50.152766\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:23:51,086 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:23:51,099 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:23:51,099 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:23:51,110 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:23:51,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:23:51,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:21.348605+00:00 (in 29.999538 seconds) gcworker stdout | 2023-01-17 06:23:51,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:21 UTC)" (scheduled at 2023-01-17 06:23:51.348605+00:00) gcworker stdout | 2023-01-17 06:23:51,350 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672727031349, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:23:51,362 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:23:51,362 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:23:51,362 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:21 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:23:52,060 [235] [DEBUG] [app] Starting request: urn:request:0bd409cd-7f33-47a3-8355-be652b8d6b42 (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:52,060 [233] [DEBUG] [app] Starting request: urn:request:32b9b5d5-ea49-48ef-96fe-3e13f55fe39e (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:23:52,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,063 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:52,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:23:52,065 [240] [DEBUG] [app] Starting request: urn:request:4ac3ae68-d310-4d75-8450-5291b6d521a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:52,065 [242] [DEBUG] [app] Starting request: urn:request:992a59a3-f019-48de-83c0-cbf598b6630d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:23:52,065 [240] [DEBUG] [app] Ending request: urn:request:4ac3ae68-d310-4d75-8450-5291b6d521a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ac3ae68-d310-4d75-8450-5291b6d521a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:23:52,066 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-registry stdout | 2023-01-17 06:23:52,066 [242] [DEBUG] [app] Ending request: urn:request:992a59a3-f019-48de-83c0-cbf598b6630d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:992a59a3-f019-48de-83c0-cbf598b6630d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:52,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:23:52,066 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:23:52,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:52,067 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:23:52,069 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:52,069 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:23:52,070 [231] [DEBUG] [app] Starting request: urn:request:d7e5f4b6-3349-4a98-89ff-d99feeb32536 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:52,070 [236] [DEBUG] [app] Starting request: urn:request:7b0dee16-07e9-440e-a62c-3d7a967e6472 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:23:52,070 [231] [DEBUG] [app] Ending request: urn:request:d7e5f4b6-3349-4a98-89ff-d99feeb32536 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d7e5f4b6-3349-4a98-89ff-d99feeb32536', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:52,071 [236] [DEBUG] [app] Ending request: urn:request:7b0dee16-07e9-440e-a62c-3d7a967e6472 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7b0dee16-07e9-440e-a62c-3d7a967e6472', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:23:52,071 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:23:52,071 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:52,071 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:23:52,071 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:23:52,071 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:52,071 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:23:52,078 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:52,078 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:23:52,087 [233] [DEBUG] [app] Ending request: urn:request:32b9b5d5-ea49-48ef-96fe-3e13f55fe39e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:32b9b5d5-ea49-48ef-96fe-3e13f55fe39e', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:52,087 [235] [DEBUG] [app] Ending request: urn:request:0bd409cd-7f33-47a3-8355-be652b8d6b42 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0bd409cd-7f33-47a3-8355-be652b8d6b42', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:23:52,087 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:52,087 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:23:52,088 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:23:52,088 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.028) nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:23:52 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.029 47 0.029) builder stdout | 2023-01-17 06:23:52,184 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:23:52.172746\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:23:54,203 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:23:54.191851\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} notificationworker stdout | 2023-01-17 06:23:55,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:23:55,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:01.636362+00:00 (in 5.989540 seconds) notificationworker stdout | 2023-01-17 06:23:55,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:05 UTC)" (scheduled at 2023-01-17 06:23:55.646354+00:00) notificationworker stdout | 2023-01-17 06:23:55,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:23:55,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 23, 55, 647281), True, datetime.datetime(2023, 1, 17, 6, 23, 55, 647281), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:23:55,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:23:55,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:23:55,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:05 UTC)" executed successfully builder stdout | 2023-01-17 06:23:56,223 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:23:56.211432\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:23:58,241 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:23:58.230096\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:00,260 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:00.249104\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:00,854 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:24:00,854 [60] [WARNING] [buildman.manager.ephemeral] Job buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8 is already in the desired state/phase (pulling) nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:24:00 +0000] "POST /buildman_pb.BuildManager/SetPhase HTTP/2.0" 200 9 "-" "grpc-go/1.41.0" (0.018 998 0.018) builder stdout | 2023-01-17 06:24:00,875 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:24:00,897 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:24:00,897 [60] [DEBUG] [buildman.jobutil.buildjob] Determined cached tag None for 2b3380f15a7268fb8eedf703065549a4b72810f927b66b16f46413fb46b9c479 nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:24:00 +0000] "POST /buildman_pb.BuildManager/DetermineCachedTag HTTP/2.0" 200 5 "-" "grpc-go/1.41.0" (0.037 1137 0.037) builder stdout | 2023-01-17 06:24:00,914 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:24:00,932 [60] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.128.2.7 (-) - - [17/Jan/2023:06:24:00 +0000] "POST /buildman_pb.BuildManager/SetPhase HTTP/2.0" 200 9 "-" "grpc-go/1.41.0" (0.035 998 0.036) builder stdout | 2023-01-17 06:24:00,968 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:00.945727\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:01,120 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:24:01,132 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:24:01,132 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:24:01,143 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:24:01,583 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'gcworker.py', 'pid': '66'} notificationworker stdout | 2023-01-17 06:24:01,632 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'notificationworker.py', 'pid': '76'} notificationworker stdout | 2023-01-17 06:24:01,636 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:24:01,636 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:05.646354+00:00 (in 4.009592 seconds) notificationworker stdout | 2023-01-17 06:24:01,637 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:01 UTC)" (scheduled at 2023-01-17 06:24:01.636362+00:00) notificationworker stdout | 2023-01-17 06:24:01,637 [76] [DEBUG] [workers.queueworker] Running watchdog. notificationworker stdout | 2023-01-17 06:24:01,637 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:01 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:24:01,828 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '235'} gunicorn-web stdout | 2023-01-17 06:24:01,838 [231] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '231'} gunicorn-web stdout | 2023-01-17 06:24:01,840 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '233'} gunicorn-web stdout | 2023-01-17 06:24:01,843 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '70'} gunicorn-web stdout | 2023-01-17 06:24:01,845 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'web:application', 'pid': '236'} builder stdout | 2023-01-17 06:24:02,300 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:02.266960\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} queuecleanupworker stdout | 2023-01-17 06:24:02,637 [78] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'queuecleanupworker.py', 'pid': '78'} logrotateworker stdout | 2023-01-17 06:24:02,983 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'logrotateworker.py', 'pid': '71'} namespacegcworker stdout | 2023-01-17 06:24:03,048 [74] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'namespacegcworker.py', 'pid': '74'} builder stdout | 2023-01-17 06:24:03,075 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'builder.py', 'pid': '60'} securityworker stdout | 2023-01-17 06:24:03,533 [87] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityworker.py', 'pid': '87'} builder stdout | 2023-01-17 06:24:04,320 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:04.308829\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} repositorygcworker stdout | 2023-01-17 06:24:04,937 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:24:04,937 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:19.346120+00:00 (in 14.408822 seconds) repositorygcworker stdout | 2023-01-17 06:24:04,937 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:04 UTC)" (scheduled at 2023-01-17 06:24:04.936835+00:00) repositorygcworker stdout | 2023-01-17 06:24:04,937 [80] [DEBUG] [workers.queueworker] Getting work item from queue. repositorygcworker stdout | 2023-01-17 06:24:04,938 [80] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 24, 4, 937725), True, datetime.datetime(2023, 1, 17, 6, 24, 4, 937725), 0, 'repositorygc/%', 50, 1, 0]) repositorygcworker stdout | 2023-01-17 06:24:04,949 [80] [DEBUG] [workers.queueworker] No more work. repositorygcworker stdout | 2023-01-17 06:24:04,950 [80] [DEBUG] [data.database] Disconnecting from database. repositorygcworker stdout | 2023-01-17 06:24:04,950 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:04 UTC)" executed successfully notificationworker stdout | 2023-01-17 06:24:05,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:24:05,646 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:15.646354+00:00 (in 9.999542 seconds) notificationworker stdout | 2023-01-17 06:24:05,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:15 UTC)" (scheduled at 2023-01-17 06:24:05.646354+00:00) notificationworker stdout | 2023-01-17 06:24:05,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:24:05,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 24, 5, 647266), True, datetime.datetime(2023, 1, 17, 6, 24, 5, 647266), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:24:05,659 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:24:05,659 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:24:05,659 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:15 UTC)" executed successfully gunicorn-secscan stdout | 2023-01-17 06:24:05,795 [219] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '219'} gunicorn-secscan stdout | 2023-01-17 06:24:05,797 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '69'} gunicorn-secscan stdout | 2023-01-17 06:24:05,799 [220] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'secscan:application', 'pid': '220'} gunicorn-registry stdout | 2023-01-17 06:24:06,304 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '68'} gunicorn-registry stdout | 2023-01-17 06:24:06,311 [230] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '230'} gunicorn-registry stdout | 2023-01-17 06:24:06,315 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '241'} gunicorn-registry stdout | 2023-01-17 06:24:06,322 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '242'} gunicorn-registry stdout | 2023-01-17 06:24:06,325 [232] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '232'} gunicorn-registry stdout | 2023-01-17 06:24:06,328 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '240'} gunicorn-registry stdout | 2023-01-17 06:24:06,332 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '243'} gunicorn-registry stdout | 2023-01-17 06:24:06,337 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '237'} gunicorn-registry stdout | 2023-01-17 06:24:06,339 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'registry:application', 'pid': '239'} builder stdout | 2023-01-17 06:24:06,340 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:06.328336\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} buildlogsarchiver stdout | 2023-01-17 06:24:06,692 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'buildlogsarchiver.py', 'pid': '61'} expiredappspecifictokenworker stdout | 2023-01-17 06:24:06,815 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '64'} gunicorn-web stdout | 2023-01-17 06:24:07,060 [231] [DEBUG] [app] Starting request: urn:request:7141cfb3-2895-4e49-8567-cf125169f5ff (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:24:07,060 [235] [DEBUG] [app] Starting request: urn:request:1f138eb5-3d96-4b6b-924f-9fcdef034a0b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:24:07,061 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,061 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,063 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:07,064 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:24:07,065 [243] [DEBUG] [app] Starting request: urn:request:0af040c1-5063-4ff0-ad53-491600471231 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:07,065 [230] [DEBUG] [app] Starting request: urn:request:ee2015f4-e440-425d-b85f-6df4e1bd5784 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:07,065 [243] [DEBUG] [app] Ending request: urn:request:0af040c1-5063-4ff0-ad53-491600471231 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0af040c1-5063-4ff0-ad53-491600471231', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-registry stdout | 2023-01-17 06:24:07,066 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-registry stdout | 2023-01-17 06:24:07,066 [230] [DEBUG] [app] Ending request: urn:request:ee2015f4-e440-425d-b85f-6df4e1bd5784 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee2015f4-e440-425d-b85f-6df4e1bd5784', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:24:07,066 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:24:07,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:24:07,066 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 162 0.002) gunicorn-web stdout | 2023-01-17 06:24:07,067 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,067 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:07,069 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:07,070 [235] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:07,071 [236] [DEBUG] [app] Starting request: urn:request:8c816a33-a50a-4af3-8b4d-649fb8370a03 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:07,071 [233] [DEBUG] [app] Starting request: urn:request:f9444b9f-4816-4859-8a6f-f8a16e7ebfde (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:07,071 [236] [DEBUG] [app] Ending request: urn:request:8c816a33-a50a-4af3-8b4d-649fb8370a03 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c816a33-a50a-4af3-8b4d-649fb8370a03', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:24:07,072 [233] [DEBUG] [app] Ending request: urn:request:f9444b9f-4816-4859-8a6f-f8a16e7ebfde (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9444b9f-4816-4859-8a6f-f8a16e7ebfde', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:24:07,072 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:24:07,072 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:07,072 [233] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 159 0.001) gunicorn-web stdout | 2023-01-17 06:24:07,072 [235] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:07,072 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:07,072 [235] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:07,081 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:07,082 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:07,091 [231] [DEBUG] [app] Ending request: urn:request:7141cfb3-2895-4e49-8567-cf125169f5ff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7141cfb3-2895-4e49-8567-cf125169f5ff', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:07,091 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.032) gunicorn-web stdout | 2023-01-17 06:24:07,091 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" gunicorn-web stdout | 2023-01-17 06:24:07,091 [235] [DEBUG] [app] Ending request: urn:request:1f138eb5-3d96-4b6b-924f-9fcdef034a0b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1f138eb5-3d96-4b6b-924f-9fcdef034a0b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:07,091 [235] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:24:07,092 [235] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:24:07 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.033 47 0.032) storagereplication stdout | 2023-01-17 06:24:07,938 [89] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'storagereplication.py', 'pid': '89'} servicekey stdout | 2023-01-17 06:24:07,969 [88] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'servicekeyworker.py', 'pid': '88'} builder stdout | 2023-01-17 06:24:08,359 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:08.347707\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} repositorygcworker stdout | 2023-01-17 06:24:08,442 [80] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositorygcworker.py', 'pid': '80'} chunkcleanupworker stdout | 2023-01-17 06:24:08,800 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'chunkcleanupworker.py', 'pid': '62'} manifestbackfillworker stdout | 2023-01-17 06:24:08,857 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'manifestbackfillworker.py', 'pid': '72'} securityscanningnotificationworker stdout | 2023-01-17 06:24:09,102 [81] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'securityscanningnotificationworker.py', 'pid': '81'} repositoryactioncounter stdout | 2023-01-17 06:24:09,310 [79] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'repositoryactioncounter.py', 'pid': '79'} globalpromstats stdout | 2023-01-17 06:24:09,758 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'globalpromstats.py', 'pid': '67'} teamsyncworker stdout | 2023-01-17 06:24:10,361 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'teamsyncworker.py', 'pid': '90'} builder stdout | 2023-01-17 06:24:10,378 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:10.366945\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} exportactionlogsworker stdout | 2023-01-17 06:24:10,435 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'exportactionlogsworker.py', 'pid': '65'} blobuploadcleanupworker stdout | 2023-01-17 06:24:10,702 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56b949c797-mg688', 'process_name': 'blobuploadcleanupworker.py', 'pid': '59'} buildlogsarchiver stdout | 2023-01-17 06:24:10,738 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run buildlogsarchiver stdout | 2023-01-17 06:24:10,739 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:40.738568+00:00 (in 29.999556 seconds) buildlogsarchiver stdout | 2023-01-17 06:24:10,739 [61] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:40 UTC)" (scheduled at 2023-01-17 06:24:10.738568+00:00) buildlogsarchiver stdout | 2023-01-17 06:24:10,739 [61] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2023, 1, 2, 6, 24, 10, 739414), False, 50, 1, 0]) buildlogsarchiver stdout | 2023-01-17 06:24:10,751 [61] [DEBUG] [__main__] No more builds to archive buildlogsarchiver stdout | 2023-01-17 06:24:10,751 [61] [DEBUG] [data.database] Disconnecting from database. buildlogsarchiver stdout | 2023-01-17 06:24:10,751 [61] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:40 UTC)" executed successfully builder stdout | 2023-01-17 06:24:11,146 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:24:11,159 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:24:11,159 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:24:11,170 [60] [DEBUG] [data.database] Disconnecting from database. builder stdout | 2023-01-17 06:24:12,398 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:12.386267\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:14,418 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:14.406017\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} notificationworker stdout | 2023-01-17 06:24:15,646 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run notificationworker stdout | 2023-01-17 06:24:15,647 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:25.646354+00:00 (in 9.999543 seconds) notificationworker stdout | 2023-01-17 06:24:15,647 [76] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:25 UTC)" (scheduled at 2023-01-17 06:24:15.646354+00:00) notificationworker stdout | 2023-01-17 06:24:15,647 [76] [DEBUG] [workers.queueworker] Getting work item from queue. notificationworker stdout | 2023-01-17 06:24:15,648 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2023, 1, 17, 6, 24, 15, 647531), True, datetime.datetime(2023, 1, 17, 6, 24, 15, 647531), 0, 'notification/%', 50, 1, 0]) notificationworker stdout | 2023-01-17 06:24:15,660 [76] [DEBUG] [workers.queueworker] No more work. notificationworker stdout | 2023-01-17 06:24:15,660 [76] [DEBUG] [data.database] Disconnecting from database. notificationworker stdout | 2023-01-17 06:24:15,660 [76] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2023-01-17 06:24:25 UTC)" executed successfully builder stdout | 2023-01-17 06:24:16,437 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:16.424936\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} securityworker stdout | 2023-01-17 06:24:16,937 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:24:16,937 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:19.537472+00:00 (in 2.599957 seconds) securityworker stdout | 2023-01-17 06:24:16,937 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:46 UTC)" (scheduled at 2023-01-17 06:24:16.936988+00:00) securityworker stdout | 2023-01-17 06:24:16,938 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:24:16,938 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:24:16,941 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:24:16,942 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:24:16,953 [87] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:24:16,956 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:24:16,956 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:46 UTC)" executed successfully builder stdout | 2023-01-17 06:24:18,457 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:18.445380\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} repositorygcworker stdout | 2023-01-17 06:24:19,346 [80] [DEBUG] [apscheduler.scheduler] Looking for jobs to run repositorygcworker stdout | 2023-01-17 06:24:19,346 [80] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:25:04.936835+00:00 (in 45.590161 seconds) repositorygcworker stdout | 2023-01-17 06:24:19,347 [80] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:19 UTC)" (scheduled at 2023-01-17 06:24:19.346120+00:00) repositorygcworker stdout | 2023-01-17 06:24:19,347 [80] [DEBUG] [workers.queueworker] Running watchdog. repositorygcworker stdout | 2023-01-17 06:24:19,347 [80] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2023-01-17 06:25:19 UTC)" executed successfully securityworker stdout | 2023-01-17 06:24:19,537 [87] [DEBUG] [apscheduler.scheduler] Looking for jobs to run securityworker stdout | 2023-01-17 06:24:19,538 [87] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:46.936988+00:00 (in 27.398996 seconds) securityworker stdout | 2023-01-17 06:24:19,538 [87] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:49 UTC)" (scheduled at 2023-01-17 06:24:19.537472+00:00) securityworker stdout | 2023-01-17 06:24:19,538 [87] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:19,538 [87] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ... securityworker stdout | 2023-01-17 06:24:19,540 [87] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:24:19,540 [87] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds securityworker stdout | 2023-01-17 06:24:19,540 [87] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:19,541 [87] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request securityworker stdout | 2023-01-17 06:24:19,541 [87] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state securityworker stdout | 2023-01-17 06:24:19,543 [87] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None securityworker stdout | 2023-01-17 06:24:19,545 [87] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', []) securityworker stdout | 2023-01-17 06:24:19,555 [87] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:19,555 [87] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop securityworker stdout | 2023-01-17 06:24:19,556 [87] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing securityworker stdout | 2023-01-17 06:24:19,556 [87] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped securityworker stdout | 2023-01-17 06:24:19,556 [87] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'. securityworker stdout | 2023-01-17 06:24:19,557 [87] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST securityworker stdout | 2023-01-17 06:24:19,557 [87] [DEBUG] [data.database] Disconnecting from database. securityworker stdout | 2023-01-17 06:24:19,557 [87] [ERROR] [workers.worker] Operation raised exception securityworker stdout | Traceback (most recent call last): securityworker stdout | File "/quay-registry/workers/worker.py", line 87, in _operation_func securityworker stdout | return operation_func() securityworker stdout | File "/quay-registry/workers/securityworker/securityworker.py", line 45, in _index_recent_manifests_in_scanner securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/__init__.py", line 40, in perform_indexing_recent_manifests securityworker stdout | self._model.perform_indexing_recent_manifests(batch_size) securityworker stdout | File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 246, in perform_indexing_recent_manifests securityworker stdout | start_index = max(end_index - batch_size, 1) securityworker stdout | TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' securityworker stdout | 2023-01-17 06:24:19,557 [87] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:49 UTC)" executed successfully builder stdout | 2023-01-17 06:24:19,759 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:19.747762\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:19,777 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:19.765594\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:19,795 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:19.783143\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,042 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.028337\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,063 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.048756\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,081 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.069570\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,099 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.087270\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,116 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.104868\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,132 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.121709\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,149 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.138685\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,166 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.155484\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,183 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.172496\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,202 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.189614\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,221 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.208325\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,237 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.226629\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,254 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.243374\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,271 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.260295\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,288 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.277122\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,305 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.294089\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,322 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.311582\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,340 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.328909\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,357 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.346099\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,374 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.363073\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,390 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.379900\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,407 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.396577\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,424 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.413352\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,441 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.429879\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,457 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.446762\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,474 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.462732\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,481 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.466889\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:20,498 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:20.486946\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:21,180 [60] [DEBUG] [buildman.manager.ephemeral] Checking for more work from the build queue builder stdout | 2023-01-17 06:24:21,196 [60] [DEBUG] [buildman.manager.ephemeral] No additional work found. Going to sleep for 10 seconds builder stdout | 2023-01-17 06:24:21,196 [60] [DEBUG] [buildman.manager.ephemeral] Writing queue metrics builder stdout | 2023-01-17 06:24:21,210 [60] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:24:21,348 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run gcworker stdout | 2023-01-17 06:24:21,349 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2023-01-17 06:24:51.348605+00:00 (in 29.999530 seconds) gcworker stdout | 2023-01-17 06:24:21,349 [66] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:51 UTC)" (scheduled at 2023-01-17 06:24:21.348605+00:00) gcworker stdout | 2023-01-17 06:24:21,349 [66] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100]) gcworker stdout | 2023-01-17 06:24:21,361 [66] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1672727061360, 1209600, True, 3, 500, 1, 0]) gcworker stdout | 2023-01-17 06:24:21,366 [66] [DEBUG] [__main__] No repository with garbage found gcworker stdout | 2023-01-17 06:24:21,366 [66] [DEBUG] [data.database] Disconnecting from database. gcworker stdout | 2023-01-17 06:24:21,366 [66] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2023-01-17 06:24:51 UTC)" executed successfully gunicorn-web stdout | 2023-01-17 06:24:22,060 [233] [DEBUG] [app] Starting request: urn:request:01d46759-0131-4dde-b704-2bfd4aaa9a8c (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:24:22,060 [231] [DEBUG] [app] Starting request: urn:request:950a3e88-738b-4fea-b719-48afeb21075b (/health/instance) {'X-Forwarded-For': '10.129.2.1'} gunicorn-web stdout | 2023-01-17 06:24:22,061 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,062 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,064 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:24:22,065 [230] [DEBUG] [app] Starting request: urn:request:7fc7d65a-351c-4509-8f0f-cbca0fcd523f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:22,065 [230] [DEBUG] [app] Ending request: urn:request:7fc7d65a-351c-4509-8f0f-cbca0fcd523f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7fc7d65a-351c-4509-8f0f-cbca0fcd523f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:24:22,065 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-registry stdout | 2023-01-17 06:24:22,066 [230] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.001) gunicorn-web stdout | 2023-01-17 06:24:22,066 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-registry stdout | 2023-01-17 06:24:22,066 [243] [DEBUG] [app] Starting request: urn:request:e74b8bed-b758-4ccf-8c49-4b8f7451f121 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-registry stdout | 2023-01-17 06:24:22,067 [243] [DEBUG] [app] Ending request: urn:request:e74b8bed-b758-4ccf-8c49-4b8f7451f121 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e74b8bed-b758-4ccf-8c49-4b8f7451f121', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '974e7d1c', 'user-agent': 'python-requests/2.27.1'} nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.002 162 0.002) gunicorn-registry stdout | 2023-01-17 06:24:22,067 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" gunicorn-web stdout | 2023-01-17 06:24:22,067 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /v1/_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:22,068 [233] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,069 [231] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost gunicorn-web stdout | 2023-01-17 06:24:22,070 [233] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:22,071 [231] [DEBUG] [app] Starting request: urn:request:3d6c4cbe-13a5-4121-b366-dc0d4c9d7609 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:22,072 [231] [DEBUG] [app] Ending request: urn:request:3d6c4cbe-13a5-4121-b366-dc0d4c9d7609 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d6c4cbe-13a5-4121-b366-dc0d4c9d7609', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:24:22,072 [231] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.002) gunicorn-web stdout | 2023-01-17 06:24:22,072 [233] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:22,073 [231] [WARNING] [py.warnings] /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings gunicorn-web stdout | warnings.warn( gunicorn-web stdout | 2023-01-17 06:24:22,073 [233] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:22,074 [236] [DEBUG] [app] Starting request: urn:request:2e69175e-d56e-4f5a-8aed-f609bd64ac3f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'} gunicorn-web stdout | 2023-01-17 06:24:22,074 [236] [DEBUG] [app] Ending request: urn:request:2e69175e-d56e-4f5a-8aed-f609bd64ac3f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e69175e-d56e-4f5a-8aed-f609bd64ac3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'https://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'python-requests/2.27.1'} gunicorn-web stdout | 2023-01-17 06:24:22,074 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.27.1" nginx stdout | 127.0.0.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.27.1" (0.001 159 0.001) gunicorn-web stdout | 2023-01-17 06:24:22,075 [231] [DEBUG] [urllib3.connectionpool] https://localhost:8443 "GET /_internal_ping HTTP/1.1" 200 4 gunicorn-web stdout | 2023-01-17 06:24:22,075 [231] [INFO] [data.database] Connection pooling disabled for postgresql gunicorn-web stdout | 2023-01-17 06:24:22,080 [233] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:22,082 [231] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1]) gunicorn-web stdout | 2023-01-17 06:24:22,091 [233] [DEBUG] [app] Ending request: urn:request:01d46759-0131-4dde-b704-2bfd4aaa9a8c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01d46759-0131-4dde-b704-2bfd4aaa9a8c', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:22,091 [233] [DEBUG] [data.database] Disconnecting from database. gunicorn-web stdout | 2023-01-17 06:24:22,091 [233] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.032 47 0.033) gunicorn-web stdout | 2023-01-17 06:24:22,092 [231] [DEBUG] [app] Ending request: urn:request:950a3e88-738b-4fea-b719-48afeb21075b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:950a3e88-738b-4fea-b719-48afeb21075b', 'remote_addr': '10.129.2.1', 'http_method': 'GET', 'original_url': 'https://10.129.2.33/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '3b590129', 'user-agent': 'kube-probe/1.24'} gunicorn-web stdout | 2023-01-17 06:24:22,092 [231] [DEBUG] [data.database] Disconnecting from database. nginx stdout | 10.129.2.1 (-) - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/2.0" 200 152 "-" "kube-probe/1.24" (0.033 47 0.033) gunicorn-web stdout | 2023-01-17 06:24:22,092 [231] [INFO] [gunicorn.access] 10.129.2.1 - - [17/Jan/2023:06:24:22 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.24" builder stdout | 2023-01-17 06:24:22,096 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.084139\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,139 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.127096\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,155 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.144802\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,172 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.161421\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,190 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.178212\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,207 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.195861\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,224 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.213150\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,243 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.230479\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,261 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.249840\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,282 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.267386\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,300 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.288120\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,317 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.306557\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,335 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.323412\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,351 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.340739\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,371 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.357609\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,388 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.377521\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,405 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.394432\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,422 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.411249\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,439 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.428134\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,456 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.444855\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,473 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.461756\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,490 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.479459\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,493 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.481037\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,510 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.499403\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,527 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.516007\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,543 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.532487\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,560 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.549131\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,576 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.565846\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,593 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.582642\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,610 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.599414\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,627 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.616192\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,645 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.633296\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,664 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.650558\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,681 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.670534\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,700 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.687363\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,718 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.706184\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,736 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.724382\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,752 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.741743\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,769 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.758401\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,786 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.775089\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,812 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.791674\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,829 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.818008\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,846 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.834960\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,863 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.852073\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,880 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.869220\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,897 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.885981\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,921 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.903151\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,940 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.927228\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,958 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.946593\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,976 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.964478\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:22,994 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.982537\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"} builder stdout | 2023-01-17 06:24:23,012 [60] [DEBUG] [buildman.orchestrator] publishing event: {"event": 2, "key": "buildman/production/building/582cef57-43a5-43d1-9ca7-1d229e4a10d8", "value": "{\"job_queue_item\": {\"id\": 1, \"body\": \"{\\\"build_uuid\\\": \\\"582cef57-43a5-43d1-9ca7-1d229e4a10d8\\\", \\\"pull_credentials\\\": null}\", \"retries_remaining\": 2}, \"max_expiration\": 1673943817, \"last_heartbeat\": \"2023-01-17 06:24:22.999829\", \"executor_name\": \"openshift\", \"execution_id\": \"582cef57-43a5-43d1-9ca7-1d229e4a10d8-dwchk\"}"}